Voxel Transformer for 3D Object Detection

In plain words

Self-driving cars and robots often measure their surroundings with a laser scanner, which returns a cloud of points. Many detectors first pack those points into a grid of small 3D cells called voxels and then run convolutions over the grid to find objects. A convolution only looks at a small neighbourhood of cells, so the detector sees little context beyond that, and an object with only a handful of points is hard to recognise.

VoTr replaces the convolutional backbone with attention, so that each occupied cell can look at other occupied cells over a wide area. Most cells are empty, so the model only looks at occupied ones and only at a capped number of them per cell. The neighbours are chosen with a near range and a far range that uses bigger steps, so the reach is large while the cost stays small. A hash table lets the GPU find those neighbours quickly.

The result is meant to be a drop-in backbone for existing voxel detectors. The authors report that it beats the convolutional versions of the same detectors on the KITTI and Waymo Open datasets.

TL;DR

• VoTr is a Transformer backbone for voxel-based 3D object detection on LiDAR point clouds §1 §3.
• It runs self-attention only over non-empty voxels, with local and dilated attention ranges capped at 48 voxels per query §3.2 §3.3 §4.1.
• A GPU hash table (Fast Voxel Query) finds the attending voxels §3.4.
• Author-reported: VoTr-SSD beats SECOND on Waymo and KITTI, and VoTr-TSD beats PV-RCNN on KITTI test moderate car tab. 1 tab. 2.
• Code is public, but the repository has no LICENSE file, so only cite it; do not copy code README.md.

Problem

• Convolutional 3D backbones have small receptive fields. In the authors' example, the largest receptive field of the last layer of a common backbone on KITTI is about 3.65 m × 3.65 m × 7.3 m, which hardly covers a car longer than 4 m §1.
• Enlarging the receptive field by changing voxel size, kernel size, stride or depth each has a cost: quantization error, cubic kernel growth, low-resolution BEV maps, or more computation §1.
• Plain Transformer attention on voxels is infeasible for two reasons. Non-empty voxels are very sparse (below 0.1% of the voxel grid on Waymo), and they are numerous (about 90k per frame on Waymo), so full self-attention is too costly §1.
• The attention range must therefore reach far while each query attends to few voxels; the authors state a target of fewer than 50 attending voxels §3.3.

Method

Overview. VoTr is a 3D backbone that outputs voxel features. Input voxels are first projected to 16 channels. Then come three sparse voxel modules, each downsampling by 2 along every axis and followed by two submanifold voxel modules that keep the same non-empty locations. The output is projected to a bird's-eye-view (BEV) map for the detection head, and the voxels can also feed a second-stage RoI refinement §3.1 fig. 2. Multi-head attention runs in all these modules §3.1.

Self-attention on sparse voxels §3.2. Let voxel i be a query with features f_i and integer index v_i, and let Ω(i) be its set of attending voxels j. Each voxel index is converted to a voxel centre p = r · (v + 0.5), where r is the voxel size. For a single head, queries, keys and values are

Q_i = f_i W_q,   K_j = f_j W_k + E_pos,   V_j = f_j W_v + E_pos   [eq. 1]
E_pos = (p_i - p_j) W_pos                                        [eq. 2]

and attention is

f_i^attend = sum over j in Omega(i) of softmax(Q_i K_j / sqrt(d)) V_j   [eq. 3]

where σ is softmax and d is not further defined in the text §3.2 eq. 3. The authors present this as a 3D counterpart of standard 2D self-attention, applied to sparse inputs, where relative coordinates supply the positional information §3.2.

Submanifold voxel module §3.2. Outputs sit exactly at the input non-empty locations. It has two sub-layers with residual connections: the multi-head attention over all attention mechanisms, then a feed-forward layer §3.2. Three changes to a standard Transformer block:

1. an extra linear projection after the feed-forward layer, for channel change;
2. batch normalization instead of layer normalization;
3. no dropout.

Sparse voxel module §3.2 eq. 4. This module creates outputs at empty locations, which is needed for downsampling. A query for an empty location has no feature, so its query is approximated from its attending features by max-pooling, Q_i = max over j in Ω(i) of f_j eq. 4. Attention is then computed with eq. 3. The first residual around self-attention is removed, because input and output differ §3.2. Other parts match the submanifold module.

Attention ranges §3.3. The attending set Ω(i) is built from two mechanisms that both pick non-empty voxels on an integer grid around the query.

• Local Attention takes every non-empty voxel within R_local = (1, 1, 1) of the query, with stride (1, 1, 1) eq. 5. The experiments use R_local = (1, 1, 1) §3.3.
• Dilated Attention takes a list of M ring-shaped ranges. Range m covers offsets between R_start and R_end with step R_stride, minus the inner range eq. 6. Later ranges have larger starts, ends and strides. The attention reaches more than 15 m, and still keeps fewer than 50 attending voxels, with a suitable list §3.3 app. B.
• Local Attention is the special case of Dilated Attention with R_start = (0,0,0), R_end = (1,1,1) and R_stride = (1,1,1) §3.3.

Fast Voxel Query §3.4 fig. 4. A dense grid holding a per-voxel index would be too large, since the dense grid has more than 10^7 cells §3.4. A per-query scan of all non-empty voxels costs O(N_sparse) per query §3.4. Fast Voxel Query uses a GPU hash table instead, with four steps:

1. Build a hash table of N_hash × 2 entries, where the first column holds hashed integer voxel indices and the second holds indices into the voxel array §3.4.
2. For each query, compute the attending indices with local and dilated ranges §3.4.
3. Look up each attending index. A miss (value −1) marks an empty voxel, which is rejected §3.4.
4. Gather the attending indices and features and run attention §3.4.

Linear probing resolves collisions and atomic operations prevent data races. Each query gets its own CUDA thread §3.4. The paper gives N_hash = 400k in experiments §4.1.

# Build the hash table: voxel index -> row in the voxel array  [§3.4]
# Input:  voxel_idx (N, 4) as [batch, z, y, x]
# Output: table (hash_size, 2) of (key, row); empty slots hold EMPTY

def build_hash_table(voxel_idx, hash_size):
    # Start with every slot empty  [§3.4]
    table = full((hash_size, 2), EMPTY)
    for row in range(N):
        # Hash the integer key of this voxel  [§3.4]
        slot = hash(voxel_idx[row]) % hash_size
        # Linear probing until a free slot is claimed  [§3.4]
        while table[slot, 0] != EMPTY:
            slot = (slot + 1) % hash_size
        table[slot] = (key_of(voxel_idx[row]), row)
    return table

# Look up one attending voxel index  [§3.4]
# Output: the row in the voxel array, or -1 if the voxel is empty

def lookup(table, key, hash_size):
    # Start at the hashed slot  [§3.4]
    slot = hash(key) % hash_size
    # Probe until the key is found or an empty slot ends the search  [§3.4]
    while table[slot, 0] != EMPTY:
        if table[slot, 0] == key:
            return table[slot, 1]
        slot = (slot + 1) % hash_size
    return -1

# One sparse voxel module, condensed  [§3.2] [eq. 4]
# Input:  voxel_feats (N, C), voxel_idx (N, 4)
# Output: new_feats (M, C_out) at downsampled, possibly empty, locations

def sparse_voxel_module(voxel_feats, voxel_idx):
    # Choose downsampled locations, stride 2 along each axis  [app. A]
    new_idx = downsample(voxel_idx, stride=2)               # (M, 4)
    # Attending voxels per new location, capped at 48  [§4.1] [§3.3]
    keys_idx = attending_rows(new_idx, local_and_dilated)   # (M, S), -1 empty
    key_feats = gather(voxel_feats, keys_idx)               # (M, S, C)
    # The query has no feature, so max-pool its attending features  [eq. 4]
    query = max_over_slots(key_feats)                       # (M, C)
    # Relative positions and positional embedding  [eq. 2]
    rel = center(new_idx) - center(keys_idx)                # (M, S, 3)
    key_feats = key_feats + rel @ W_pos                     # (M, S, C)
    # Multi-head attention; empty slots are masked  [eq. 3]
    attended = multi_head_attention(query, key_feats, key_feats, keys_idx < 0)
    # No residual around attention; feed-forward keeps its residual  [§3.2]
    out = batch_norm(attended + feed_forward(attended))
    # Extra linear projection to the output channels  [§3.2]
    return linear(out)

Two details in the sparse module are the paper's choices but not stated in full in the text: the exact feed-forward layout and where the batch normalization sits. The condensed block above follows the text in §3.2.

Hyperparameters

Architecture and attention §4.1 unless noted.

Dilated Attention ranges on KITTI app. B tab. 9. Written as paper values (R_start, R_end, R_stride) per axis. Modules 2–4 share one setting, as do 5–7 and 8–9. The text says the same configuration is used for VoTr-SSD and VoTr-TSD on KITTI.

The KITTI config in code has the same numbers, in order votr_ssd.yaml.

Training §4.1. Paper and code differ; see Paper vs code.

Common settings §4.1: Adam optimizer; cosine annealing for learning-rate decay; other settings as in SECOND and PV-RCNN for data augmentation. On Waymo, 20% of training frames are sampled uniformly and evaluation uses the full validation set, following PV-RCNN §4.1.

Results

Author-reported. Setup §4.1: Waymo Open Dataset (798 training sequences, 202 validation sequences) §4.1; KITTI with the train/val and test splits §4.1. Baselines: SECOND and PV-RCNN, with the paper's own rows for PV-RCNN(*) re-implemented with the official code tab. 1. The VoTr-SSD and VoTr-TSD rows are the paper's models.

Headline numbers the paper reports:

• Waymo LEVEL_1 3D mAP: VoTr-SSD 68.99 (SECOND 67.94 re-implemented); VoTr-TSD 74.95 (PV-RCNN 70.3 as reported; PV-RCNN* 74.06 re-implemented) tab. 1.
• KITTI test, moderate car AP_3D: VoTr-SSD 78.25 (SECOND 75.96); VoTr-TSD 82.09 (PV-RCNN 81.43) tab. 2.
• KITTI test, hard car AP_3D: VoTr-TSD 79.14, which the authors call higher than all listed prior work tab. 2 §4.3.
• KITTI val, moderate car AP_3D: VoTr-SSD 78.27 (SECOND 76.48); VoTr-TSD 84.04 (PV-RCNN 83.69) tab. 3.
• Waymo 50m–Inf LEVEL_1 3D mAP: VoTr-SSD 42.08 (SECOND 40.36); VoTr-TSD 51.09 (PV-RCNN 42.2) tab. 1.
• Efficiency: VoTr-SSD runs at 14.65 Hz versus SECOND at 20.73 Hz, about 20 ms added latency; VoTr-SSD has 4.8M parameters versus 5.3M for SECOND tab. 7 tab. 8 §4.4.

Table 1. Waymo Open Dataset, vehicle, 202 validation sequences tab. 1. * marks rows re-implemented with the official code.

Table 2. KITTI test set, car, AP_3D (%) with 40 recall positions tab. 2.

Table 3. KITTI val split, car, AP_3D (%) with 11 recall positions tab. 3.

Ablations on KITTI val (moderate car AP_3D unless noted) §4.4.

Paper vs code

Code: https://github.com/PointsCoder/VOTR at commit 5ce1b80890989f9babeec9eef75a241338f89a73, read as cloned on 2026-10-19. The owner account PointsCoder shows the display name Jiageng Mao, the first author. The README calls the repository "a reproduced repo" built on OpenPCDet README.md. Differences found:

1. Training schedule, KITTI. Paper: batch 32 (SSD) and 16 (TSD), learning rate 0.01, 80 epochs, 8 V100 GPUs §4.1. Code: VoTr-SSD uses 4 per GPU, learning rate 0.003, 100 epochs votr_ssd.yaml; VoTr-TSD uses 2 per GPU, learning rate 0.01, 80 epochs votr_tsd.yaml. The README says the KITTI SSD config is for a single GPU README.md.
2. Training schedule, Waymo. Paper: learning rate 0.003 for both, with 60 epochs for SSD and 80 for TSD §4.1. Code: SSD uses 80 epochs at 0.003 votr_ssd.yaml; TSD uses 60 epochs at 0.01 votr_tsd.yaml. The paper's epoch counts are swapped and its learning rate for TSD differs.
3. Scheduler and optimizer. Paper: Adam with cosine annealing §4.1. Code: adam_onecycle with Adam betas (0.9, 0.99), weight decay 0.01, one-cycle schedule with pct_start 0.4, div_factor 10, momentum cycling between 0.95 and 0.85, and gradient clipping at 10 votr_ssd.yaml [code: tools/train_utils/optimization/init.py#L19-L22] [code: tools/train_utils/optimization/init.py#L50-L53]. The paper does not name a one-cycle schedule, and the one-cycle class itself was not read here.
4. Relative position embedding. Paper: E_pos is a linear map of p_i − p_j eq. 2. Code: the key embedding is a 1 × 1 convolution from 3 to C channels followed by ReLU votr_backbone.py. Relative coordinates are taken as key minus query, which is the reverse sign of eq. 2 votr_backbone.py and votr_backbone.py.
5. Extra normalization and activation. The paper says the input projection is a linear layer to 16 channels §4.1, and that the output has an extra linear projection §3.2. Code adds batch normalization and ReLU after the input projection votr_backbone.py, and after the output projection votr_backbone.py.
6. Residual across submanifold pairs. The paper describes residuals around the sub-layers inside each module §3.2. The code also adds the input of each sparse module's two submanifold blocks back to their output votr_backbone.py. The paper does not mention this.
7. Query positional term. Eq. 1 has no position term in the query, which matches the configs: USE_NO_QUERY_COORDS is True, so the query position projection is skipped votr_ssd.yaml votr_backbone.py.
8. Sparse query. The max-pool that forms the query eq. 4 is taken before the positional embedding is added to the keys votr_backbone.py and votr_backbone.py.
9. Attention range convention. The paper writes the ranges as closed intervals §3.3 eq. 6. The CUDA kernel loops over offsets from 0 up to but not including the end value, with step equal to the stride, and mirrors them in all sign combinations build_attention_indices_gpu.cu. Under this reading the code's first range [0, 2) covers offsets ±1, which matches the paper's R_local = (1, 1, 1). Read as a closed interval it would cover ±2.
10. Waymo configuration. Paper: only the KITTI ranges are given app. B. Code: the Waymo SSD config uses different ranges and SIZE 27 in the first sparse module votr_ssd.yaml.
11. Hash function. The paper gives no hash function. The code uses the integer key modulo the table size build_mapping_gpu.cu. A murmur-style hash is defined in the same file but not used by hash build_mapping_gpu.cu.
12. Voxel centres. The paper gives p = r(v + 0.5) §3.2. The code adds the point cloud range minimum as an offset votr_backbone.py. Relative differences are unchanged, and absolute coordinates are only used in the query term, which is disabled.
13. Global Attention. Commented out in the paper source §3.3. The code has a global-attention variant in its V2 classes, but the configs use the V1 class [code: pcdet/models/backbones_3d/init.py#L1-L6].

No discrepancy was found in the attention ranges of the KITTI configs: the dilated ranges in app. B tab. 9 match the first three KITTI sparse modules and submanifold blocks in votr_ssd.yaml.

Pitfalls

• Paper text and tables disagree in places. The text says the KITTI moderate gains of 2.29% and 0.66% are on the val split, but those numbers come from the test table tab. 2, and the val gains are 1.79% and 0.35% tab. 3 §4.3.
• Waymo gains in the text do not match Table 1. The text gives VoTr-SSD gains of 1.05% (SECOND) and VoTr-TSD gains of 3.26% over PV-RCNN. Table 1 gives 1.05 for SSD, but 74.95 − 70.3 = 4.65 for TSD, and the 3.26 figure matches no row. Its distance-band gains of 3.37% and 4.83% for VoTr-TSD also match no row, since 73.36 − 69.2 = 4.16 and 51.09 − 42.2 = 8.89 §4.2 tab. 1.
• Numerical stability of the sparse module. A downsampled location with no non-empty voxel in its range has every key masked. The code masks empty slots with key_padding_mask votr_backbone.py. I did not check how that case is handled, and the paper is silent on it.
• Memory and epochs. The README says reproduction needs 32 GB of GPU memory and more than 60 epochs on Waymo to converge README.md. Reducing the attention size is suggested as a workaround and may hurt accuracy README.md.
• Run-to-run variance. The README says KITTI results are quite unstable and suggests running several times README.md.
• No checkpoints. The authors do not release trained weights README.md.
• Hash table size. The code comment says 400k is "5x max_voxels" votr_ssd.yaml. That holds for Waymo's 80k training cap, but the KITTI test cap is 40k, so it is 10x there kitti_dataset.yaml.
• Scaling in eq. 3. The paper writes √d without defining d. The code uses PyTorch's multi-head attention module with 4 heads votr_backbone.py, so the usual per-head scaling applies. This is inferred from the code.
• Sign and offset of relative coordinates. See Paper vs code, items 4 and 12.
• Licence of the code. The repository has no LICENSE file. Code is cited, not copied. Pseudocode above is written from the paper, not the code.
• Figure numbering. Figures are numbered here in the order they appear in the LaTeX source; they were not checked against the compiled PDF.

Lineage

• Builds on: SECOND, the sparse-convolution voxel backbone it replaces §3.1 §4.1; PV-RCNN, the second-stage framework for VoTr-TSD §4.1.
• Attention ideas borrowed from Zaheer et al., BigBird, for the local and dilated patterns §3.3.
• Compared with Voxel R-CNN's dense voxel grids for neighbour lookup, whose memory cost the paper calls too high §3.4.
• Compared against: PointPillars, PV-RCNN, Voxel R-CNN and point-based detectors on KITTI and Waymo tab. 1 tab. 2 tab. 3.
• Superseded by: not stated in the content of this wiki.