---
kind: concept
slug: set-abstraction
title: Set abstraction
summary: >-
  The set abstraction layer of point-based 3D detectors: pick key points, group the points around each one, and pool one feature per key point. The variable part is how key points are chosen. Plain FPS, feature-distance FPS and foreground-weighted FPS (SASA) are compared. Gains are author-reported on KITTI and nuScenes.
tags: [point-cloud, 3d-object-detection, sampling]
papers: ["2201.01976"]
related: [farthest-point-sampling, object-detection]
status: draft
updated_at: 2026-11-15
---

## Overview

A set abstraction (SA) layer is the down-sampling stage of a PointNet++-style
point network. It selects a smaller set of key points from its input, groups
the points around each key point, runs a shared MLP and max-pools, so every
key point gets one feature [arxiv:2201.01976 §3]. The layers are stacked. In
3DSSD, level 1 keeps 4096 key points and levels 2 and 3 keep 512 + 512 and
256 + 256, where each sum is an S-FPS half and an FPS half
[arxiv:2201.01976 tab. 5] [arxiv:2201.01976 §3.3.1]. In PointRCNN the key-point
counts are 4096, 1024, 256 and 64 [arxiv:2201.01976 tab. 6].

The key-point rule decides which points survive each level. Plain
farthest-point sampling (FPS) spreads the picks over the scene, and in outdoor
scans most of those picks land on background such as ground
[arxiv:2201.01976 §1]. Earlier detectors recovered the dropped points with
feature propagation (FP) layers, which the paper says add memory and compute
cost [arxiv:2201.01976 §1]. The variants below change the sampling rule. The
MLP and the pooling stay the same.

Only one card is linked to this slug so far, so the variants below are the
ones that paper describes.

## Variants

### Plain FPS sampling

The sampler picks points by spread alone, with no per-point scores or
labels. SASA's first SA level uses it, because that level's input is the raw
point cloud and has no foreground scores yet [arxiv:2201.01976 §3.3.1].

### Feature-distance FPS (F-FPS, 3DSSD)

3DSSD samples its levels 2 and 3 with F-FPS, which measures distance between
feature vectors rather than between coordinates [arxiv:2201.01976 §3.3.1].
The paper says this costs quadratically in the number of points, because it
needs pairwise feature distances [arxiv:2201.01976 §4.3.3].

### Score-weighted FPS (S-FPS, SASA)

SASA adds a segmentation head to each SA layer and uses its output in the
sampler. The steps, as the paper defines them:

- **Score.** A two-layer MLP reads each point's features at that layer and
  outputs a foreground score p in [0, 1] through a sigmoid [arxiv:2201.01976
  §3.1] [eq. 1].
- **Supervision.** A point is foreground if it lies inside any ground-truth 3D
  box, and background otherwise. Each SA layer has a cross-entropy loss, and
  the total is a weighted sum over layers [arxiv:2201.01976 §3.1] [eq. 2]. This
  variant therefore needs labelled boxes.
- **Sampling.** The first key point is the highest-scoring point. Each later
  key point maximises p raised to γ, times its distance to the nearest key
  point already chosen [arxiv:2201.01976 §3.2] [alg. 1] [eq. 3]. γ = 0 gives
  plain FPS for the later picks. A very large γ approaches a top-K selection on
  scores [arxiv:2201.01976 §3.2].
- **Weights.** The paper's ablation favours γ = 1 [arxiv:2201.01976 §4.4]
  [tab. 3]. Loss weights for 3DSSD are 0.01 at level 2 and 0.1 at level 3, and
  level 1 is not supervised [arxiv:2201.01976 §3.3.1]. For PointRCNN they are
  0.001, 0.01 and 0.1 at levels 2 to 4 [arxiv:2201.01976 §3.3.2].
- **Placement.** In 3DSSD, S-FPS replaces F-FPS for half of the key points at
  levels 2 and 3, and plain FPS samples the other half. The other sampling
  settings stay as they were [arxiv:2201.01976 §3.3.1] [fig. 3]. In PointRCNN
  it is used at levels 2 to 4 [arxiv:2201.01976 §3.3.2].
- **Reported gains (author-reported).** On the KITTI validation split, moderate
  car AP for 3DSSD with fusion sampling rises from 83.46 (F-FPS) to 85.76
  (S-FPS, γ = 1), and point recall from 96.65 to 97.65 [arxiv:2201.01976
  tab. 3]. On the KITTI test set, 3DSSD moderate car AP rises from 79.57 to
  82.16 [arxiv:2201.01976 tab. 1]. On nuScenes validation, NDS rises from 56.4
  to 61.0 [arxiv:2201.01976 tab. 2].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Raw points at the first level, or no per-point labels | Plain FPS | The first level has no scores [arxiv:2201.01976 §3.3.1], and the score head is trained from ground-truth boxes [arxiv:2201.01976 §3.1]. |
| Labelled 3D boxes, and FPS would keep mostly background | S-FPS | This is the case the paper targets [arxiv:2201.01976 §1]. Its gains are reported on KITTI and nuScenes [arxiv:2201.01976 tab. 1] [tab. 2] [tab. 3]. |
| Keeping the 3DSSD layout with feature-distance sampling | F-FPS | It is the baseline that S-FPS replaces at levels 2 and 3 [arxiv:2201.01976 §3.3.1] [tab. 3]. The paper's cost argument against it is the quadratic growth [arxiv:2201.01976 §4.3.3]. |
| Latency is a constraint | Measure on your own hardware | The paper reports 36 ms for 3DSSD with SASA against 38 ms without, on one V100 with OpenPCDet [arxiv:2201.01976 tab. 1] [§4.3.3]. These numbers come from the authors' setup. |
| Trying γ above 1 | Check numerical precision | The paper reports γ = 100 [arxiv:2201.01976 tab. 3]. At that γ, p^γ underflows in float32 for most scores, which the card's [Pitfalls](/papers/2201.01976) section explains. |

## Implementation sketch

The layer core is the same whichever sampler is used. Only the call to the
sampler changes.

```pseudocode
# Set abstraction layer  [arxiv:2201.01976 §3]
# Input:  coords (N, 3), feats (N, C), num_keys M
# Output: key_xyz (M, 3), key_feats (M, C_out)

def set_abstraction(coords, feats, num_keys, sampler):
    # Choose key points. The sampler may read per-point scores  [§3.2]
    key_idx = sampler(coords, feats, num_keys)     # (M,)
    key_xyz = coords[key_idx]                      # (M, 3)
    # Group the points around each key point  [§3]
    groups = group(coords, feats, key_xyz)         # (M, K, 3 + C)
    # Shared MLP, then max-pool over each group  [§3]
    key_feats = max_pool(mlp(groups), axis=1)      # (M, C_out)
    return key_xyz, key_feats
```

For S-FPS, the sampler first computes a score per point, then scores the
candidates [arxiv:2201.01976 eq. 1] [eq. 3]:

```pseudocode
# Score-weighted sampler  [arxiv:2201.01976 alg. 1] [eq. 3]
# Input:  scores (N,) in (0, 1); gamma; num_keys M
# Output: key_idx (M,)

def s_fps_sampler(coords, scores, gamma, num_keys):
    # First key point: highest foreground score  [alg. 1]
    first = argmax(scores)
    key_idx = [first]
    chosen = zeros(N, dtype=bool)
    chosen[first] = True
    min_dist = norm(coords - coords[first], axis=1)
    for _ in range(num_keys - 1):
        # Weighted distance, chosen points excluded  [eq. 3]
        weighted = scores ** gamma * min_dist
        weighted[chosen] = -inf
        nxt = argmax(weighted)
        key_idx.append(nxt)
        chosen[nxt] = True
        min_dist = minimum(
            min_dist, norm(coords - coords[nxt], axis=1))
    return key_idx
```

The released code and the paper differ on the distance used here. See
Contradictions.

## Contradictions

- **Distance in S-FPS.** Algorithm 1 and eq. 3 use the Euclidean distance
  [arxiv:2201.01976 alg. 1] [eq. 3]. The released CUDA kernel squares it, so at
  γ = 1 the code ranks by p times d², not p times d. The card's
  [Paper vs code](/papers/2201.01976) section records this.
- **Labels around boxes.** The paper gives two labels, foreground inside any
  ground-truth box and background otherwise [arxiv:2201.01976 §3.1]. The
  released code adds an ignored band around each box, which its loss leaves
  out. The card's [Paper vs code](/papers/2201.01976) section records this.

## Papers

- [SASA](/papers/2201.01976): adds a foreground head and score-weighted FPS to
  the SA layers of 3DSSD and PointRCNN, with author-reported gains on KITTI and
  nuScenes.
