Paper · 2022-01-06 · cs.CV
SASA: Semantics-Augmented Set Abstraction for Point-based 3D Object Detection
SASA adds a binary foreground head to the set abstraction layers of point-based 3D detectors and samples key points with a foreground-weighted farthest point sampling (S-FPS) instead of plain FPS, so more object points survive down-sampling. Author-reported: 3DSSD + SASA reaches 82.16% moderate car AP on KITTI test (baseline 79.57%) and 61.0 NDS on nuScenes val (baseline 56.4).
Object detectionFarthest-point samplingSet abstraction3d-object-detectionpoint-cloudlidarsampling
arXivPDFCodeMarkdown10 min read
AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Point-based 3D detectors read LiDAR scans as raw points. Each down-sampling step keeps a spread-out subset of points, and most of the scene is ground and other background. Plain farthest-point sampling keeps many of those background points and can drop points on small or distant objects, which then hurt detection.
SASA adds a small side network that guesses, for every point, how likely it is to belong to an object. The down-sampling step then favours points with high scores, while still spreading the kept points across the scene. The rest of the detector stays the same, so the module can be added to more than one point-based detector.
The point of the change is better features for the points that matter for detection. The paper reports gains on two driving datasets and says the module reaches results close to voxel-based detectors.
TL;DR
SASA is a plug-in set abstraction layer for point-based 3D detectors. It predicts a foreground score per point with a small segmentation head, trained from ground-truth boxes, and uses it in a score-weighted farthest point sampling (S-FPS). The paper integrates it into 3DSSD and PointRCNN. Gains are author-reported on KITTI and nuScenes. The released code differs from the paper in the loss normalisation, an ignore band around boxes, and the distance used in sampling (see Paper vs code).
Problem
Farthest-point sampling (FPS) picks points that cover the scene, so in outdoor scans most picks land on background such as ground §1. Foreground points on small objects, such as pedestrians, can be discarded, and the box head then has less object information to work with §1. Earlier point-based detectors recover dropped points with feature propagation (FP) layers, which the paper says add memory and compute cost §1. The paper's assumptions: the detector has set abstraction (SA) layers and ground-truth 3D boxes for training §3.1.
Method
Each SASA layer is a normal PointNet++ SA layer with two additions: a point segmentation module and a sampling step that uses its scores. Input coordinates X and features F go to the segmentation module, which outputs a foreground score per point. S-FPS then selects key points from X using those scores. The key points go through the usual grouping, MLP and max-pooling of the SA layer §3 fig. 2.
-
Foreground score. A two-layer MLP reads each point's input features at that SA layer and outputs a score in [0, 1] through a sigmoid §3.1 eq. 1.
p_i = \sigma(\mathcal{M}_k(f^{(l_k)}_i)) eq. 1
-
Foreground labels and loss. A point is foreground if it lies inside any ground-truth 3D box, otherwise background §3.1. Each SA layer has a cross-entropy loss, and the total is a weighted sum over the layers §3.1 eq. 2.
\mathcal{L}_{seg} = \sum_{k=1}^{m}{ \frac{\lambda_k}{N_k} \cdot \sum_{i=1}^{N_k}{ \text{CE}(p_i^{[k]}, \hat{p}_i^{[k]}) } } eq. 2
Here N_k is the number of input points and λ_k the layer weight §3.1.
-
Semantics-guided FPS. The first key point is the input point with the highest foreground score alg. 1. Each later key point maximises the product of the score raised to γ and the distance to the nearest key point already chosen §3.2 eq. 3 alg. 1. Setting γ = 0 gives plain FPS for the later picks, and very large γ approaches a top-K selection on scores §3.2.
\tilde{d}_i = p_i^\gamma \cdot d_i eq. 3
# Semantics-guided farthest point sampling (S-FPS) [alg. 1] [eq. 3]
# Input: coords (N, 3); fg_scores (N,) in (0, 1); gamma; num_keys (M)
# Output: key_idx (M,)
def s_fps(coords, fg_scores, gamma, num_keys):
# distance to the nearest chosen key point, starts at +inf [alg. 1]
min_dist = full(N, inf) # (N,)
chosen = zeros(N, dtype=bool) # (N,)
key_idx = []
for step in range(num_keys):
if step == 0:
# first key point: highest foreground score [alg. 1]
best = argmax(fg_scores)
else:
# weighted distance p^gamma * d, unchosen points only [eq. 3]
weighted = fg_scores ** gamma * min_dist # (N,)
weighted[chosen] = -inf
best = argmax(weighted)
key_idx.append(best)
chosen[best] = True
# refresh each point's distance to the nearest key point [alg. 1]
new_dist = norm(coords - coords[best], axis=1) # (N,)
min_dist = minimum(min_dist, new_dist)
return key_idx- Where it is placed. In 3DSSD, SASA is applied at levels 2 and 3 of the backbone. Each of these layers uses fusion sampling: S-FPS replaces the feature-distance FPS (F-FPS) for half of the key points, and plain FPS samples the other half §3.3.1 fig. 3. In PointRCNN, the SASA layers are levels 2 to 4, and the backbone otherwise keeps the original structure, including FP layers §3.3.2 fig. 3.
- Training. The segmentation losses are added to the detector loss §3.1 eq. 2. Layer weights for 3DSSD are 0.01 and 0.1, and for PointRCNN 0.001, 0.01 and 0.1, across the SASA levels §3.3.1 §3.3.2.
Other points of the method, as stated: S-FPS keeps more diverse foreground points, concentrates key points in high-score areas, is less sensitive to distant outliers, and is permutation-invariant because its start point is set by score rather than input order §3.2. The paper says S-FPS avoids the pairwise feature-distance computation of F-FPS, whose cost grows quadratically with the number of points §4.3.3.
Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| Segmentation head | 2-layer MLP, sigmoid output | §3.1 eq. 1 | Hidden widths are not in the paper; see code row below |
| Segmentation head widths (code) | 3DSSD: 32 (level 2), 64 (level 3); PointRCNN: 32, 64, 128 (levels 2 to 4) | 3dssd_sasa.yaml pointrcnn_sasa.yaml | Each head: one hidden conv with batch norm and ReLU, then a one-channel output pointnet2_modules.py |
| Balance factor γ | 1 | §4.4 tab. 3 | γ = 1 is the best setting in the paper's ablation; code default 1.0 3dssd_sasa.yaml |
| Segmentation loss weights λ_k, 3DSSD | level 2: 0.01; level 3: 0.1 | §3.3.1 | Level 1 is not supervised (it cannot give meaningful scores) §3.3.1 |
| Segmentation loss weights λ_k, PointRCNN | level 2: 0.001; level 3: 0.01; level 4: 0.1 | §3.3.2 | Same values in pointrcnn_sasa.yaml |
| Segmentation loss weights λ_k, nuScenes 3DSSD | 0.001, 0.01, 0.1 for levels 2 to 4 | not stated in paper | Code only: 3dssd_sasa.yaml |
| 3DSSD SA sizes | level 1: 4096; level 2: 512 + 512; level 3: 256 + 256 key points | tab. 5 §3.3.1 | The paper says other sampling settings stay unchanged §3.3.1 |
| PointRCNN SA sizes | 4096, 1024, 256, 64 key points | tab. 6 | |
| Optimiser, 3DSSD (KITTI) | Adam, one-cycle schedule, peak learning rate 0.01, 80 epochs | §4.2.1 | Batch size stated as 16 in total on four GPUs §4.2.1; code differs, see Paper vs code |
| Optimiser, PointRCNN (KITTI) | Adam, one-cycle schedule, peak learning rate 0.01, 80 epochs | §4.2.2 | Batch size not stated. Code uses 0.005 pointrcnn_sasa.yaml |
| Optimiser, 3DSSD (nuScenes) | 20 epochs, total batch 16 on eight GPUs | §4.2.1 | Learning rate not stated in paper; code 0.002 3dssd_sasa.yaml |
| nuScenes input | Key frame plus earlier sweeps within 0.5 s; 0.1 m voxels; 16,384 key-frame and 49,152 earlier-frame voxels, one random point each; 65,536 points; features: xyz, reflectance, timestamp | §4.2.1 | |
| KITTI input points | not stated in paper | Code samples 16,384 points per training and test cloud 3dssd_sasa.yaml | |
| Augmentation, KITTI 3DSSD | GT-AUG; flip along X; global scaling in [0.9, 1.1]; global rotation in [-π/4, π/4] | §4.2.1 | Box-wise augmentation: paper says "random permutation, scaling and rotation"; code config lists box noise with rotation ±π/3 and no scaling 3dssd_sasa.yaml |
| 3D NMS, 3DSSD (inference) | threshold 0.01 | §4.2.1 | nuScenes config uses 0.2 3dssd_sasa.yaml |
Results
Author-reported. Setups: KITTI car class, test split via the official server §4.3.1 tab. 1; nuScenes validation split §4.3.2 tab. 2; KITTI val split, trained on the train split, for ablations §4.4 tab. 3 tab. 4. Baselines are the same detector without SASA. Inference time is measured with OpenPCDet on one V100 §4.3.3.
| Setting | Metric | Baseline | This paper | Source |
|---|---|---|---|---|
| KITTI test, car, 3DSSD | AP, easy (IoU 0.7) | 88.36 | 88.76 | tab. 1 |
| KITTI test, car, 3DSSD | AP, moderate | 79.57 | 82.16 | tab. 1 |
| KITTI test, car, 3DSSD | AP, hard | 74.55 | 77.16 | tab. 1 |
| KITTI test, car, 3DSSD | time per sample (ms) | 38 | 36 | tab. 1 |
| nuScenes val, 3DSSD | NDS | 56.4 | 61.0 | tab. 2 |
| nuScenes val, 3DSSD | mAP | 42.6 | 45.0 | tab. 2 |
| KITTI val, car, 3DSSD, fusion sampling | AP, moderate | 83.46 (F-FPS) | 85.76 (S-FPS, γ = 1) | tab. 3 |
| KITTI val, car, 3DSSD, fusion sampling | point recall | 96.65 (F-FPS) | 97.65 (S-FPS, γ = 1) | tab. 3 |
| KITTI val, car, PointRCNN | AP, hard | 80.45 | 82.40 | tab. 4 |
| KITTI val, car, PointRCNN | mAP | 84.75 | 85.72 | tab. 4 |
| KITTI val, car, 3DSSD | mAP | 85.73 | 87.02 | tab. 4 |
Other numbers the paper reports: the per-layer foreground rates in the appendix, for example 35.23% for S-FPS against 9.09% for F-FPS at level 2 of 3DSSD tab. 5. PointRCNN's level 4 point recall rises from 39.62 to 96.36 with S-FPS, with 64 key points tab. 6. The paper states that its KITTI test results are comparable with voxel-based detectors, which are listed in tab. 1 §4.3.1.
Paper vs code
Searched: the abstract links https://github.com/blakechen97/SASA, which is the authors' repository (an OpenPCDet fork with the SASA configs). It is Apache-2.0 and was cloned at commit cd79f60e923242590b64cb0cc70203a524e7e9a7. The paper's arXiv v1 date is 2022-01-06; the commit date is 2022-02-18, so the code may be newer than the paper.
Discrepancies found:
- Sampling distance. Algorithm 1 and eq. 3 use the Euclidean distance alg. 1 eq. 3. The CUDA kernel uses the squared Euclidean distance, and the weight is applied to that value sampling_gpu.cu. With γ = 1 the code therefore weights by p times d², not p times d.
- Ignore band. The paper labels points as foreground inside any GT box and background otherwise §3.1. The code's loss grows each GT box by the extra width of 1.0 m per dimension (centre unchanged) box_utils.py. Points inside the grown box but outside the GT box get label -1 and are left out of the loss.
set_ignore_flagis True in all three configs 3dssd_sasa.yaml pointrcnn_sasa.yaml 3dssd_sasa.yaml. The ignore band is in code only. Label assignment: loss_utils.py. - Loss normaliser. Eq. 2 divides each layer's sum by N_k, the number of input points §3.1 eq. 2. The code divides by the number of non-ignored points in the layer, clamped to at least 1 loss_utils.py. The variable name
pos_normalizeris misleading. The loss itself is binary cross-entropy on logits, applied to the score before the sigmoid loss_utils.py. - PointRCNN learning rate. Paper: peak 0.01 §4.2.2. Code: LR 0.005 pointrcnn_sasa.yaml.
- 3DSSD batch size. Paper: total batch 16 on four GPUs §4.2.1. Code:
BATCH_SIZE_PER_GPU: 83dssd_sasa.yaml. Which setting produced the reported numbers is not stated. - nuScenes architecture. The paper does not describe a separate nuScenes backbone. The code's nuScenes 3DSSD has four SA levels with other sizes, a dilated grouping setup and different radii 3dssd_sasa.yaml. The nuScenes learning rate (0.002) and NMS threshold (0.2) are also not in the paper 3dssd_sasa.yaml 3dssd_sasa.yaml.
- Box augmentation. The paper lists random permutation, scaling and rotation of boxes §4.2.1. The KITTI 3DSSD config has random box noise with rotation ±π/3, location noise of 1.0 m in x and y, and no scaling; no permutation step is listed 3dssd_sasa.yaml.
Matches checked: score definition (sigmoid of the head output, raised to γ) pointnet2_modules.py; 3DSSD and PointRCNN layer weights; γ = 1 for all SASA layers 3dssd_sasa.yaml pointrcnn_sasa.yaml; the 80-epoch one-cycle Adam setup; and the 3DSSD NMS threshold of 0.01 for KITTI 3dssd_sasa.yaml.
The scores used at SA layer k are predicted from the output features of layer k-1, which are the features fed to layer k. This matches eq. 1 pointnet2_backbone.py pointnet2_modules.py.
Pitfalls
- Ignore band and label definition. The paper's text gives two labels. The code uses three (foreground, background, ignore), with the ignore band set by a 1.0 m box extension. An implementation that follows only the paper will train on a different set of points.
- Squared versus Euclidean distance. Implementations that follow Algorithm 1 literally use d, while the released kernel uses d². At γ = 1 the two rank candidates differently, so the reported gains depend on which the implementer chooses. Matching the released code is the way to reproduce the paper's numbers, but the paper's notation is different.
- Underflow at large γ. The weight is p^γ, with p from a sigmoid. At γ = 100 (one of the paper's ablation rows tab. 3), p^γ underflows to zero in float32 for any p below about 0.35. The kernel then floors the weight at 1e-12 sampling_gpu.cu. Under that floor, every point with p below about 0.76 has the same weight, so only the distance orders those points. The paper reports the γ = 100 result but does not discuss this.
- First key point. The first key point is the highest-score point, not a fixed or random one alg. 1. This keeps S-FPS independent of input order (the paper's permutation claim §3.2). Ties in score are broken by argmax order, which the paper does not specify.
- Fusion sampling indices. At 3DSSD levels 2 and 3, the S-FPS and FPS halves are concatenated in the output. The next layer reads them by index ranges 3dssd_sasa.yaml. Off-by-one errors in these ranges change which points are foreground-sampled.
- Level 1 has no scores. The first SA layer samples with plain FPS because its input is raw points §3.3.1. The scores for the second level come from the first level's features.
- Cost. The paper claims S-FPS avoids the quadratic pairwise feature distance of F-FPS §4.3.3. The per-step cost is still O(N) per key point, the same as FPS. The timing (36 ms) was measured on one V100 with OpenPCDet §4.3.3.
- Dataset-specific settings. The nuScenes input recipe (voxel sampling into 65,536 points) and the box augmentation are described only partly. Reproducing nuScenes numbers needs the config file, not the paper alone.
- Author-reported numbers. The ablations train on the KITTI train split and evaluate on the val split, with the authors' reproduced 3DSSD as baseline §4.1.1 §4.4. They are not the same comparison as the test-server table tab. 1.
Lineage
- Builds on: the PointNet++ set abstraction layer, used unchanged apart from the added segmentation head and the sampler §3.
- Baselines and detectors it plugs into: 3DSSD (with its F-FPS fusion sampling) and PointRCNN §3.3.
- Compared against: FPS, F-FPS, and top-K selection on foreground scores §3.2 §4.4; voxel-based and point-based detectors listed in tab. 1.
- Related sampling work named in the paper: differentiable sampling methods and heuristic sampling, such as the feature-distance F-FPS of 3DSSD §2.
Source and license
Adapted from “SASA: Semantics-Augmented Set Abstraction for Point-based 3D Object Detection” by Chen Chen, Zhe Chen, Jing Zhang, Dacheng Tao (arXiv:2201.01976), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/blakechen97/SASA at commit cd79f60e9232 (Apache-2.0). Any quoted code is shown under that license.