wikillm

Paper · 2022-10-13 · cs.CV

SWFormer: Sparse Window Transformer for 3D Object Detection in Point Clouds

SWFormer is a pure Transformer 3D object detector for LiDAR point clouds. It voxelises points into a bird's-eye grid, groups non-empty voxels into sparse windows bucketed by length, shifts windows once per scale, fuses five scales and adds voxel diffusion to create features near object centres. Author-reported: 73.36 L2 mAPH on the Waymo Open Dataset test set, above the other listed methods.

Shifted-window attentionBird's-eye-view (BEV) representationMulti-scale feature fusionHeatmap regression3d-object-detectionlidarpoint-cloudtransformerwaymo-open-datasetvoxel-diffusion

arXivPDFMarkdown11 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

A self-driving car's LiDAR returns a cloud of points. Many detectors squash the cloud into a flat top-down picture and run image-style networks on it. SWFormer keeps the picture sparse instead. Only the grid cells that contain points are kept, and they are grouped into small square patches. A Transformer looks inside each patch, so cells that share a patch can talk to each other.

Patches hold very different numbers of occupied cells, so the patches are sorted into groups of similar fill and processed together. The network then works at several resolutions, and once per scale it shifts the patch grid so that neighbouring patches exchange information. Object centres are often empty in a LiDAR scan. A final step decides which cells are probably foreground and spreads their features into the empty cells around them, so that a box can be predicted near the centre of each object.

The point of the design is to avoid dense convolutions over empty space while still letting distant parts of the scene interact. The authors report the best Waymo test-set score of the methods they list.

TL;DR

SWFormer is a convolution-free, Transformer-only 3D object detector for LiDAR point clouds. Non-empty voxels are grouped into sparse windows and bucketed by length. Each scale has one window shift, five scales are fused top-down, and a voxel-diffusion step fills cells near object centres before a CenterNet-style head regresses boxes. Author-reported: 73.36 L2 mAPH on the Waymo test set, above every method in the paper's test table tab. 3. Pedestrian L2 AP there is not the top score. Several details needed to re-implement it are not stated, so check the Pitfalls first.

Problem

Voxelised 3D detectors use dense convolutions over a bird's-eye grid that is mostly empty, so cost grows with the grid size §2.1. Sparse convolutions avoid this but cannot connect sparsely placed features without extra strided layers, and they need custom GPU ops that matmul-oriented hardware such as TPUs handles poorly §1, §2.1. Window-based Transformers such as Swin scale better, but shifting windows at every layer is costly on sparse data §1, §3.2. Detection also suffers because LiDAR returns are sparse around object centres, which are where anchors or centre heatmaps should sit §3.4.

Method

Overview §3.1, fig. 1. Points carry per-frame voxel features and a frame timestamp offset, following PointPillars and RSN §3.1. A PointNet-style embedding with dynamic voxelisation produces sparse voxel features §3.1. These go through a five-scale hierarchical sparse window encoder §3.2, a top-down multi-scale fusion network §3.3, voxel diffusion §3.4, and a CenterNet-style head with box regression §3.5. Vehicle and pedestrian heads can sit on different feature scales fig. 1.

Sparse windows and buckets §3.2, fig. 2. The bird's-eye voxel grid is cut into non-overlapping windows of fixed size H×W. A window can hold between 0 and HW non-empty voxels §3.2. The windows are grouped into at most k buckets B_0 … B_k. Windows in bucket B_i are zero-padded to length HW/2^i, and padded tokens are masked in attention §3.2. Self-attention within a window shares keys and values across the window's query voxels §3.2.

Strided window partition §3.2, fig. 2. At the start of each scale, the grid is downsampled by a stride. For each stride window, the non-empty voxel closest to the window centre is kept, with a deterministic tie-break. If every voxel in the stride window is empty, the output voxel is empty. No max or average pooling is used §3.2.

Block §3.2, fig. 3, eq. 1, eq. 2. Each block has N Transformer layers on the windows, then one shifted window partition, then M more layers, so N+M layers and a single shift. Layers use post-norm (LayerNorm after the attention and after the MLP) and standard sine/cosine absolute positional encodings §3.2, eq. 1, eq. 2. The strides are {1, 2, 4, 16, 32} for the five scales, and the window size stays fixed across scales §3.2.

Pseudocode
# SWFormer block  [§3.2] [eq. 1] [eq. 2]
# Input:  windows (num_windows, bucket_len, d)  with d = 128 [§4.2]
# Output: windows (num_windows, bucket_len, d)

def swformer_block(windows, pad_mask, num_layers_n, num_layers_m):
    # Stage 1: N post-norm Transformer layers, windows fixed  [eq. 1]
    tokens = windows + sine_cosine_encoding(windows)
    for _ in range(num_layers_n):
        tokens = layer_norm(tokens + msa(tokens, pad_mask))
        tokens = layer_norm(tokens + mlp(tokens))
    # Stage 2: one shifted window partition, then M layers  [eq. 2]
    tokens, pad_mask = shift_window(tokens, pad_mask)
    tokens = tokens + sine_cosine_encoding(tokens)
    for _ in range(num_layers_m):
        tokens = layer_norm(tokens + msa(tokens, pad_mask))
        tokens = layer_norm(tokens + mlp(tokens))
    return tokens

Multi-scale fusion §3.3, fig. 4, eq. 3, eq. 4. Starting from the coarsest scale P_5, each fused map is built from the next coarser one. The coarser map is upsampled to the stride of P_i and concatenated with P_i. A one-layer SWFormer block then produces the fused map §3.3, eq. 3, eq. 4. Upsampled features are copied only to locations where P_i is non-empty, so the fused map keeps the sparsity of P_i §3.3, fig. 4.

Pseudocode
# Top-down multi-scale fusion  [§3.3] [fig. 4] [eq. 3] [eq. 4]
# Input:  encoder_maps, list P_0 ... P_5 (fine to coarse)
# Output: fused_maps, list with the same strides

def fuse_scales(encoder_maps):
    fused = [None] * len(encoder_maps)
    # The coarsest map is used as it is  [eq. 3]
    fused[-1] = encoder_maps[-1]
    for i in reversed(range(len(encoder_maps) - 1)):
        # Upsample the coarser fused map to P_i's stride  [§3.3]
        up = upsample_to_stride(fused[i + 1], encoder_maps[i])
        # Copy upsampled features only to non-empty cells of P_i  [§3.3]
        up = keep_nonempty(up, encoder_maps[i])
        # Concatenate and run a one-layer SWFormer block  [eq. 4]
        fused[i] = swformer_1layer(concat(encoder_maps[i], up))
    return fused

Voxel diffusion §3.4, fig. 5, eq. 5. A foreground/background classifier gives every voxel a score in [0, 1] §3.4. Voxels with a score above γ are scattered onto a dense bird's-eye grid. A k×k max pooling then spreads the valid cells to their neighbours. The new cells start with zero features, and the diffused features go through a few Transformer layers §3.4, fig. 5. The foreground loss is a two-class focal loss averaged over valid voxels [§3.4, eq. 5].

Pseudocode
# Voxel diffusion  [§3.4] [fig. 5]
# Input:  voxels (V, d), scores (V,) in [0, 1]
# Output: dense bird's-eye map (H, W, d)

def voxel_diffusion(voxels, scores, diffusion_k, gamma):
    # Keep foreground voxels with score above gamma (0.05)  [§3.4] [§4.2]
    foreground = scores > gamma
    # Scatter the kept voxels onto the dense grid  [§3.4]
    dense_bev = scatter_to_grid(voxels[foreground], foreground)
    # k x k max pooling of the valid mask adds neighbour cells  [§3.4]
    diffused_mask = max_pool2d(valid_mask(dense_bev), diffusion_k)
    # New cells get zero-initialised features  [§3.4]
    return zero_fill(dense_bev, diffused_mask)

Box regression §3.5, eq. 6 to eq. 10. The head is a modified CenterNet head that follows RSN §3.5. The heatmap loss is a penalty-reduced focal loss per class, with ε, α and β as stated §3.5, eq. 6. Boxes have seven parameters: centre offsets relative to the voxel centre, length, width, height and heading §3.5. Heading uses a bin loss §3.5, eq. 7. The other parameters use smooth L1 §3.5, eq. 8. An IoU loss is added on voxels whose ground-truth heatmap value is above δ_1 §3.5, eq. 9. The total loss is a weighted sum over classes §3.5, eq. 10.

Pseudocode
# Decoding  [§3.5]
# Input:  per-voxel heatmap h (V,), boxes (V, 7)
# Output: detections, no NMS

def decode(heatmap, boxes, delta_2):
    # Local maxima of the heatmap, found by max pooling  [§3.5]
    peaks = heatmap == max_pool2d(heatmap)
    # Drop voxels with a heatmap below delta_2 (0.1)  [§3.5] [§4.2]
    keep = peaks & (heatmap >= delta_2)
    # No non-maximum suppression follows  [§3.5]
    return boxes[keep]

Hyperparameters

Values in the paper. Where a value is not stated, the table says so.

NameValueWhereNotes
Voxel size0.32 m in x and y; infinite in z§4.2dynamic voxelisation §3.1
Min points per GT box (training)5§4.2boxes with fewer points are ignored
Voxel embedding MLP2 layers, 128 channels§4.2
Transformer channels, heads, MLP ratio128, 8, 2§4.2
Stochastic depth survival0.6§4.2
Window size H×W10×10 (given as "e.g.")§3.2the value used in training is not stated explicitly
Scale strides{1, 2, 4, 16, 32}§3.2five scales
Number of buckets knot stated§3.2
N and M (layers before and after shift)not stated§3.2blocks per scale not stated
Foreground cutoff γ0.05§3.4 §4.2
Diffusion kernel knot stated for the final modeltab. 4 fig. 5ablation reference k=9 tab. 4; figure example k=5
Heatmap focal ε, α, β1e-3, 2, 4§3.5 eq. 6
Box-loss heatmap threshold δ_10.2§4.2vehicle and pedestrian
Decoding threshold δ_20.1§4.2 §3.5vehicle and pedestrian
Regression targets per frame (cap)1024 vehicle, 800 pedestrian§4.2sorted by GT heatmap value
Loss weights λ_1, λ_2200, 10§4.2 eq. 10λ_1 on segmentation, λ_2 on heatmap
Intensity and elongationnormalised with tanh§4.2
Augmentation: yaw rotationuniform in [−π, π], probability 0.74§4.2
Augmentation: flip along yprobability 0.5§4.2
Augmentation: scalingfactor in [0.95, 1.05)§4.2
Augmentation: point dropprobability 0.05§4.2
OptimiserAdam§4.2betas and ε not stated
Epochs128§4.2
Learning rate1e-3, cosine decay§4.2
Warm-up8 epochs from 5e-4§4.2
Batch sizenot stated§4.2
Weight decaynot stated§4.2
Hardware32 TPUv3 cores§4.2
Frames1 (1f) or 3 (3f)tab. 1"3f" is read from table labels; the frame count is not defined in the text

Results

Author-reported. Setup: Waymo Open Dataset, 798 train, 202 validation and 150 test scenes §4.1. Test results come from the official server. Results with test-time augmentation or ensembles are excluded §4.3.

SettingMetricBaselineThis paperSource
Test set, all classesmAPH/L2PVRCNN++ (two-stage) 71.24; RSN_3f 69.70; SST_TS_3f (two-stage) 72.9473.36tab. 3
Test set, vehicleAP/APH 3D L2PVRCNN++ 73.86/73.47; RSN_3f 71.90/71.6075.02/74.65tab. 3
Test set, pedestrianAP/APH 3D L2SST_TS_3f 76.65/73.14; RSN_3f 70.70/67.8075.87/72.07tab. 3
Validation, vehicle3D L2 AP/APHPVRCNN++ 70.6/70.2; RSN_3f 69.5/69.171.1/70.6 (3f)tab. 1
Validation, pedestrian3D L2 AP/APHSST_TS_3f 75.9/72.4; RSN_3f 68.3/63.774.8/71.1 (3f)tab. 1
Validation, vehicle 1-frame3D L2 AP/APHRSN_1f 66.0/65.5; SST_1f 65.5/65.169.2/68.8 (1f)tab. 1
Validation, large vehicle3D Large L1 APRSN_3f 45.251.5tab. 2
Validation, vehicle IoU 0.83D L1 APRSN_3f 46.447.5tab. 2
Latency, single frameXLA, Nvidia T4PointPillars about 100 ms (own impl.)43 ms; about 20 ms with fused kernels§4.3

Other results:

  • Voxel diffusion ablation, 3D L1 AP for vehicle and pedestrian: k=1 (no diffusion) 72.13 and 79.23; k=9 reaches 78.50 and 82.45 tab. 4. The text gives drops of 6.37 and 3.22 §4.4.
  • Scales, 3D L1 AP for vehicle and pedestrian: 1 scale 74.96 and 81.24; 5 scales 79.36 and 82.91 tab. 5. Two scales already add 2.72 vehicle AP §4.4.
  • Window shift, 3D L1 AP for vehicle and pedestrian: no shift 76.74 and 81.19; shift 79.36 and 82.91 tab. 5.
  • Extra window shifts (scales 1 and 2), vehicle and pedestrian: 79.17 and 82.36, against 79.36 and 82.91 for the default app. A, tab. 6. The authors also report a 10% training slowdown app. A.
  • Joint detection and segmentation, Waymo segmentation labels, about 14% of frames: mIoU 52.19 on validation and 46.82 on test, "without much tuning" app. C, tab. 7, fig. 8, fig. 9.

Paper vs code

No official code. Searched: links in the paper and its TeX (only Waymo dataset references and internal experiment links, no repository); arXiv metadata (no code field). GitHub repository search for "SWFormer" returned only third-party repositories (none from the authors); "SWFormer" restricted to the waymo-research organisation and "sparse window transformer waymo" returned nothing. Hence no code citations.

Pitfalls

  • Text and table numbers do not match. §4.3 says SWFormer is 0.42 APH/L2 above PVRCNN++ on validation; tab. 1 gives 70.6 vs 70.2, i.e. 0.40. §4.3 says 6.35 AP over RSN for large vehicles; tab. 2 gives 51.5 vs 45.2, i.e. 6.3. §4.3 says "0.12 APH/L2" over SST_3f; tab. 1 gives 70.6 vs 68.1 (2.5). The text does not say which row the 0.12 refers to. Use the tables.
  • Pedestrian is not best everywhere. On the validation set, SST_TS_3f has higher pedestrian 3D L2 (72.4 APH vs 71.1). On the test set, SST_TS_3f has higher pedestrian L2 AP (76.65 vs 75.87). The paper does not claim a pedestrian win there.
  • Frame stacking is under-specified. "3f" and "1f" appear only in table labels. The text does not give the frame count, how frames are merged, or the timestamp encoding beyond citing PointPillars and RSN §3.1.
  • Bucket rule missing. k, the rule for assigning a window to a bucket, and what happens when a window has more voxels than its bucket holds are not stated §3.2. For a 10×10 window, HW/2^i is not an integer for i ≥ 3 (100/8 = 12.5), and no rounding rule is given.
  • Stride jump. The strides are 1, 2, 4, 16, 32 §3.2. The paper does not explain why 8 is skipped.
  • Receptive-field figures are in grid cells. §3.2 states that a 10×10 window at stride 32 covers 320×320 and one shift covers 480×480 "on the original BEV voxel grid". Those are cell counts. The derived metric size, using 0.32 m voxels, is about 102 m and 154 m. This conversion is not in the paper.
  • Block depth missing. N and M, and the number of blocks per scale, are not stated §3.2. The fusion step uses a "1-layer SWFormer block" §3.3. The block is defined as N+M layers with one shift §3.2. The text does not say whether the fusion block has a shift.
  • Channels after concatenation. Fusion concatenates P_i with the upsampled map eq. 4, which doubles the channel count if both have 128 channels. The paper does not say whether a projection follows.
  • Diffusion details. The text says voxels get "zero-initialised features" from "k×k max pooling", which does not say whether pooling acts on values or on a validity mask. The number of Transformer layers after diffusion is not stated. The diffusion kernel k for the final model is not stated either; the ablation uses k=9 as reference and fig. 5 uses k=5 as an example.
  • Loss normalisation. The text defines N for the heatmap loss as the number of boxes in class c §3.5. It does not define N for the box loss eq. 9. Segmentation focal-loss α and γ are not stated §3.4. The heading bin count is not stated. The IoU loss form is not stated, only that it follows a cited work §3.5.
  • Ground-truth heatmap. The compiled text does not give the ground-truth heatmap formula. It refers to RSN §3.5. The TeX source contains a commented-out draft formula. It is not part of the paper and is not used here.
  • Decoding. The max-pool kernel size and any top-k cap are not stated §3.5.
  • Heads. The number of layers in the detection head is not stated. The vehicle and pedestrian heads may sit on different scales fig. 1.
  • Training details. Batch size, weight decay, Adam betas and ε, and frame sampling are not stated §4.2. Warm-up is described only as a number of epochs from a stated starting rate.
  • Latency comparison. The 43 ms figure is for XLA on a T4, and the PointPillars baseline uses the authors' own implementation §4.3. Treat it as author-reported.
  • Attention maps. The authors note that foreground attention scores are mostly 0 or 1 app. B, fig. 7. This is an observation, not a tested claim.

Lineage

  • Builds on: Swin Transformer (shifted windows and the hierarchy); RSN (the CenterNet-style head and loss setup, and the frame timestamp offset) §1, §3.1, §3.5.
  • Compared against: Voxel Transformer (VoTr-TSD) tab. 1; SST, PVRCNN++, CenterPoint and PointPillars tab. 1, tab. 3.
  • Superseded by: not stated.

Source and license

Adapted from “SWFormer: Sparse Window Transformer for 3D Object Detection in Point Clouds” by Pei Sun, Mingxing Tan, Weiyue Wang, Chenxi Liu, Fei Xia, Zhaoqi Leng, Dragomir Anguelov (arXiv:2210.07372), used under CC0 1.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.