Group-Free 3D Object Detection via Transformers

In plain words

Finding 3D objects in a point cloud, which is a set of unordered points in space, usually works in two steps. First the method decides which points belong to each candidate object, using a fixed rule such as "the points inside this box" or "the points that voted for this centre". Then it summarises only those points. When the rule picks the wrong points, the object description is wrong too.

This paper drops the grouping step. Each candidate object looks at all the points in the scene, and an attention mechanism learns how much each point should count for that object. The candidates are refined in several stacked layers. After each layer the current box estimate is fed back as a position hint, and the predictions from every layer are merged at the end. On two indoor benchmarks the authors report higher scores than earlier methods.

TL;DR

• Group-Free 3D detects 3D boxes from point clouds without a hand-made point-grouping step: each candidate's feature is computed by attention over all points §1 §3.2.
• A stack of transformer decoder layers refines each candidate, and each layer's position encoding is built from the box predicted by the previous layer §3.2.
• At test time the boxes from all stages are pooled and filtered with NMS at IoU 0.25 §3.2.
• Author-reported (best run / average of 25 trials): 67.3 / 66.3 mAP@0.25 and 48.9 / 48.5 mAP@0.5 on ScanNet V2 with 6 layers and 256 candidates tab. 1; 63.0 / 62.6 and 45.2 / 44.4 on SUN RGB-D tab. 2.
• The released code differs from the paper in several loss and encoding details. Read "Paper vs code" before reimplementing.

Problem

Point-based 3D detectors assign each point to an object candidate with a hand-crafted grouping rule: points inside a 3D box proposal, points inside a frustum, or points that vote for the same centre §1 §2. Wrong assignments degrade detection §1 fig. 1. Grid and voxel methods avoid grouping but suffer quantisation errors and the memory and compute cost of 3D convolutions §2.

Applying a 2D Transformer detector directly to point clouds gives significantly lower performance, according to the authors §2. With DETR's setup on ScanNet V2 for 400 epochs, the paper reports 39.6 mAP@0.25 and 21.4 mAP@0.5 tab. 10.

The method assumes an indoor point cloud as input and axis-aligned boxes for ScanNet V2, and oriented boxes for SUN RGB-D §4.1. Both benchmarks are evaluated with mean average precision at IoU 0.25 and 0.5 §4.1.

Method

The network has three parts: a backbone that gives every point a feature, a sampler that picks object candidates, and stacked attention modules that compute and refine a feature for each candidate from all points §3.

1. Backbone §3. A PointNet++ encoder-decoder §3 (4 set-abstraction and 2 feature-propagation layers §4.2) gives C-channel features for 1024 seed points at 2x resolution §3. Channel count C is not stated in the paper; the code uses 288 detector.py.
2. Candidate sampling §3.1. The paper's default is k-closest-points sampling (KPS). Each point is classified as a candidate. During training a point is positive if it lies inside a ground-truth box and is among the k points closest to the box centre. At inference the top-scoring points become the candidates. Farthest point sampling (FPS) is the alternative §3.1 tab. 3. A KPS variant with non-maximal suppression (radius 0.05) is also described §3.1, but the released code has no such branch (see Paper vs code).
3. Initial query. The candidate's point feature is projected to serve as the decoder query detector.py. The paper does not say how queries are initialised §3.2.
4. Decoder layer §3.2 fig. 3. Each layer runs multi-head self-attention among the candidates, then multi-head cross-attention from the candidates to all points, then a feed-forward network. Equations 1 and 2 define the attention eq. 1 eq. 2. Self-attention eq. 3 and cross-attention eq. 4 reuse this form with different inputs.
5. Spatial encoding §3.2. The box (x, y, z, l, h, w) of each candidate, taken from the previous stage, goes through a learned embedding for self-attention. The point coordinates (x, y, z) go through a separate learned embedding for cross-attention. Each layer has its own embeddings detector.py.
6. Prediction heads §3.3. Every decoder stage has a head: a shared two-layer MLP followed by linear outputs for objectness, box class, centre offset, size class and size offset §3.3. The centre is predicted as an offset from the candidate point modules.py.
7. Loss §3.3. Each stage's loss is a weighted sum of the five terms eq. 5. The loss is averaged over stages eq. 6, and the sampler's loss is added eq. 7.
8. Inference §3.2. The boxes from all stages are pooled and filtered with NMS at IoU 0.25 §3.2 eval_avg.py.

# Group-free detection pass  [§3] [§3.1] [§3.2]
# Input:  points (B, N, 3)
# Output: per-stage boxes and class scores

def detect(points):
    # Encode every point; keep 1024 seed points at 2x resolution  [§3]
    seed_xyz, seed_features = backbone(points)  # (B, 1024, 3), (B, C, 1024)

    # Score each seed as a candidate centre  [§3.1]
    scores = sigmoid(candidate_head(seed_features))  # (B, 1024)

    # Keep the top-scoring seeds as the O object candidates  [§3.1] [tab. 1]
    candidate_ids = topk(scores, num_candidates)  # (B, O)
    candidate_xyz = gather(seed_xyz, candidate_ids)  # (B, O, 3)
    queries = gather(seed_features, candidate_ids)  # (B, C, O)

    # First box prediction from the candidates alone  [§3.3]
    centre, size = predict_head(queries, base_xyz=candidate_xyz)
    stage_outputs = [(centre, size, scores)]

    for layer in decoder_layers:
        # Encode the current box as a position hint  [§3.2]
        box_code = concat(centre, size)  # (B, O, 6)

        # Self-attention among candidates, then cross-attention to all
        # points, each with its own learned position embedding  [eq. 3-4]
        queries = layer(queries, seed_features, box_code, seed_xyz)

        # Predict again; the centre offset is taken from the candidate point
        # (see Paper vs code)  [§3.3] [code: models/detector.py#L221-L224]
        centre, size = predict_head(queries, base_xyz=candidate_xyz)
        stage_outputs.append((centre, size, scores))

    return stage_outputs


# At test time, pool the boxes of all stages, then apply NMS at IoU 0.25
# [§3.2] [code: eval_avg.py#L26]
def final_boxes(stage_outputs):
    pooled = concat_stages(stage_outputs)
    return nms(pooled, iou_threshold=0.25)

Key equations. Multi-head attention, where W_h, V_h, Q_h and U_h are the output, value, query and key projections of head h eq. 1 §3.2:

\text{Att}(\mathbf{q}_{i}, \{\mathbf{p}_{k}\}) = \sum_{h=1}^{H} W_h \Big( \sum_{k=1}^{K} A^{h}_{i,k} \, V_h \mathbf{p}_k \Big) eq. 1

A^{h}_{i,k} = \frac{\exp[(Q_h \mathbf{q}_{i})^T (U_h \mathbf{p}_k)]}{\sum_{k'=1}^{K} \exp[(Q_h \mathbf{q}_{i})^T (U_h \mathbf{p}_{k'})]} eq. 2

Per-stage loss, with the five terms and weights \beta_1 = 0.5, \beta_2 = 0.1, \beta_3 = 1.0, \beta_4 = 0.1, \beta_5 = 0.1 §3.3:

\mathcal{L}^{(l)}_{\text{decoder}} = \beta_1 \mathcal{L}^{(l)}_{\text{obj}} + \beta_2 \mathcal{L}^{(l)}_{\text{cls}} + \beta_3 \mathcal{L}^{(l)}_{\text{center\_off}} + \beta_4 \mathcal{L}^{(l)}_{\text{sz\_cls}} + \beta_5 \mathcal{L}^{(l)}_{\text{sz\_off}} eq. 5

\mathcal{L}_{\text{decoder}} = \frac{1}{L} \sum_{l=1}^{L} \mathcal{L}^{(l)}_{\text{decoder}}, \qquad \mathcal{L} = \mathcal{L}_{\text{decoder}} + \mathcal{L}_{\text{sampler}} eq. 6 eq. 7

Figures. The paper has seven figures, all described here and none reproduced. fig. 1 shows wrong point assignments made by RoI pooling and voting. fig. 2 is the overall pipeline: backbone, sampler, stacked attention modules §3. fig. 3 shows one attention module. fig. 4 shows qualitative results per decoder stage on both datasets §4.7. fig. 5 visualises cross-attention weights per stage; the lower stages attend to the surrounding points regardless of geometry, and higher stages focus more on the object's geometry §4.7. fig. 6 and fig. 7 are supplementary qualitative results on ScanNet V2 and SUN RGB-D app. A2.

Hyperparameters

Values in the code are the defaults of train_dist.py unless noted. Where the paper does not state a value, the table says so.

Results

Author-reported. Setup: ScanNet V2 and SUN RGB-D validation sets, PointNet++ backbone unless noted, mAP at IoU 0.25 and 0.5 §4.1 §4.3. Each cell gives the best run, with the average of 25 trials in brackets tab. 1 tab. 2. The paper states that baselines are reported as best runs and that its own averages are taken over 25 trials §4.3.

ScanNet V2 tab. 1

The paper's ScanNet V2 headline is 67.3 mAP@0.25 and 48.9 mAP@0.5 with the same PointNet++ backbone as previous methods. The paper states this is 2.8 and 5.5 points above the previous best with the same backbone. With 12 decoder stages the gap at IoU 0.5 grows to 6.3 §4.3. The VoteNet row uses MMDetection3D numbers rather than the official paper's, which reported 46.8 and 24.7 on ScanNet V2 §4.3 fn. 2.

SUN RGB-D tab. 2

On SUN RGB-D the paper compares only with methods that take point clouds alone, and reports that its model beats all of them at both thresholds §4.3. imVoteNet also uses RGB images tab. 2.

Ablations (ScanNet V2, 6 layers, 256 candidates, average of 25 trials §4.4)

The paper says the RoI-pooling baseline uses max-pooling by default app. A1.2 tab. 11. The per-category AP tables for ScanNet V2 and SUN RGB-D are in the appendix tab. 12 to tab. 15 app. A2.

Inference speed (ScanNet V2, batch size 1, a single Titan XP GPU §4.5) tab. 9

The paper estimates the cost of one attention module at 0.95 GFLOPs with 256 candidates and 1024 output points, which it calls light compared to the backbone §4.5.

Paper vs code

Official code: https://github.com/zeliu98/Group-Free-3D, MIT License, copyright Ze Liu, one of the paper's authors. The code was found through the abstract's link. Checked at commit ef8b7bb5c3bf5b49b957624595dc6a642b6d0036 (2021-06-02). The discrepancies below were found by reading train_dist.py, models/, eval_avg.py and the README. Each has both citations.

1. Loss terms and weights. The paper's per-stage loss has five terms and weights 0.5, 0.1, 1.0, 0.1, 0.1 §3.3 eq. 5. The code's default objectness weight is 0.1, not 0.5 train_dist.py. The box loss also has a heading classification term (weight 0.1) and a heading regression term (weight 1), which the paper's list omits loss_helper.py. The total loss is multiplied by 10, averaged over the L + 1 heads (sampler plus decoder), and the sampler term has weight 0.8 loss_helper.py train_dist.py. Equations 6 and 7 have none of these factors eq. 6 eq. 7.
2. SUN RGB-D loss weights in the README. The README's SUN RGB-D command sets the objectness weight to 0.4 and the sampler weight to 0.2 [README: SUN RGB-D usage]. Neither value appears in the paper.
3. Positive assignment for KPS. The paper ranks points by plain distance to the box centre §3.1. The code divides the offset by the box size on each axis before ranking loss_helper.py. Seeds outside the box get a penalty of 100, so they are picked only when a box has fewer than k seeds inside it. Such filler seeds can belong to another box and stay positive; only background seeds are cleared afterwards loss_helper.py loss_helper.py.
4. Spatial encoding. The paper adds the refined encoding to the output feature of each stage and feeds it to the next stage §3.2. The code adds the encoding inside each attention layer: to the query in self-attention, and to the query and key in cross-attention transformer.py. The encoding comes from the previous stage's box detector.py. It does not enter the residual stream.
5. Cross-attention values. Equation 1 uses V_h p_k without any position term eq. 1. The code also adds the point position embedding to the values transformer.py.
6. Centre regression anchor. The paper says each stage predicts a refined location §3.2. The code computes the centre as an offset from the initial candidate point at every stage, not from the previous stage's box detector.py modules.py.
7. Attention scaling. Equation 2 has no scaling factor eq. 2. The code scales the query by head dimension to the power -0.5 multi_head_attention.py multi_head_attention.py.
8. KPS-NMS. The paper describes KPS with NMS at radius 0.05 §3.1 and reports it in tab. 3. The released code only accepts fps or kps train_dist.py detector.py. The KPS-NMS result cannot be reproduced from this code.
9. Ensemble. The paper says the ensemble uses the predictions of all stages §3.2. The evaluation code concatenates the query-generation (sampler) head with all decoder stages for the all_layers_ ensemble eval_avg.py. The paper does not say whether the sampler head is included.
10. Learning-rate groups. The paper gives the attention-module learning rate as 1/10 (ScanNet V2) or 1/20 (SUN RGB-D) of the backbone's app. A1.1. The code applies the decoder rate to every parameter whose name contains "decoder". This covers the layers, the projections and the spatial-encoding embeddings, but not the prediction heads train_dist.py.
11. Hyperparameters the paper does not state. The README sets smooth-L1 deltas that the paper does not give: ScanNet V2 centre 0.04 and size 0.111; SUN RGB-D size 0.0625, heading 0.04 and centre 0.1111 [README: ScanNet usage] [README: SUN RGB-D usage].
12. SUN RGB-D weight decay. The paper gives 1e-7 app. A1.1. The README uses 1e-8 [README: SUN RGB-D usage].
13. Reported averages. The README's averages, marked with *, differ from the paper's averages of 25 trials. Examples: ScanNet V2 L6 O256 is 66.2 and 48.4 in the README, against 66.3 and 48.5 in the paper. L12 O512 at w2× is 68.8 and 52.3 in the README, against 68.6 and 51.8 in the paper [README: ScanNet usage] tab. 1. Treat the paper's numbers as author-reported and expect small differences when reproducing.
14. Input size. §3 says the backbone receives "N points (i.e. 2048)" §3. The appendix and the code use 50k points for ScanNet V2 app. A1.1 train_dist.py. The 2048 in the code is the first set-abstraction output backbone_module.py.

No discrepancy was found in the backbone layout or the KPS default. The paper and code both use 6 decoder layers, 256 candidates, KPS with k = 4, and AdamW.

Pitfalls

• Input size. The paper says "N points (i.e. 2048)" in §3 and 50k points for ScanNet V2 in the appendix §3 app. A1.1. Implement 50k (or 20k for SUN RGB-D) as the input and 2048 as the first set-abstraction output.
• Unstated architecture sizes. The paper does not give the channel count, the number of attention heads, the FFN width or the dropout. The code uses 288 channels, 8 heads, FFN 2048 and dropout 0.1 detector.py train_dist.py.
• Post-norm residual blocks. The code applies LayerNorm after each residual addition transformer.py. The paper does not say whether the layer is pre-norm or post-norm.
• Query initialisation. The decoder query is a projection of the candidate's own point feature detector.py. The paper does not say how queries are initialised.
• Attention cost. Each decoder layer computes attention from O candidates to every seed point. Shapes from the code give an O × 1024 attention map per head per layer detector.py. The paper's speed figure covers only 256 candidates and 1024 output points §4.5.
• Gradient clipping. The paper says "gradnorm_clip" is used without a value app. A1.1. The code default is 0.1 over all parameters train_dist.py train_dist.py.
• Batch size and hardware. Batch size is not stated in the paper. The code default is 8 per GPU train_dist.py. The README says 4 GPUs are used by default. The speed numbers used batch size 1 on a Titan XP with Ubuntu 16.04, Python 3.6, CUDA 10.1 and PyTorch 1.3.1 §4.5.
• Variance. Algorithm randomness is large, according to the README and the paper's footnote §4.3 fn. 1. Compare averages across trials, not the best run tab. 1 §4.4.
• Ensemble gains depend on the dataset. The ensemble adds 3.8 mAP@0.25 on SUN RGB-D and almost nothing on ScanNet V2 tab. 7. The paper suggests this is because SUN RGB-D point clouds from real depth sensors have more missing parts §4.4. That is the authors' hypothesis, not a tested result.
• Build requirements. The PointNet++ CUDA operators must be compiled with init.sh init.sh [README: Install]. The README pins PyTorch 1.3 and CUDA 10.1 [README: Requirements].

Lineage

• Builds on: the Transformer decoder and attention §1 (Vaswani et al. 2017 is cited as the attention source), and DETR for transformer-based 2D detection §2 §4.6, see DETR. Also PointNet++ as the backbone §3 and the VoteNet-style sampling and head design §3.1 §3.3.
• Compared against: VoteNet, MLCVNet, H3DNet, HGNet, GSDN, 3D-MPA and imVoteNet on ScanNet V2 and SUN RGB-D tab. 1 tab. 2; DETR under matched backbone and heads §4.6 tab. 10; RoI-pooling and voting grouping in the same network §4.4 tab. 8.
• Superseded by: not known from content already in this wiki.