3D Object Detection for Autonomous Driving: A Survey

In plain words

A self-driving car has to know where other cars, pedestrians and cyclists are in 3D: how far away they are, how big they are, and which way they face. This survey sorts the research on that task. Methods are grouped by what they read: camera images, laser scans (LiDAR), or both together.

Cameras are cheap and rich in colour, but a flat picture does not directly tell you distance. Laser scans give distance directly, but they are sparse and have no fixed order, which makes them awkward for standard neural networks. Combining the two is promising but hard, because the two sensors see the world in different ways. Cameras also struggle at night and in rain, while laser scanners are expensive.

The authors also re-ran fifteen published detectors under one setup to compare speed and to see which kind of error hurts most. Their finding is that getting the position of a box right matters most. Many of the numbers come from the original papers, which used different hardware and settings, so they are not all directly comparable.

TL;DR

A survey of 3D object detection for autonomous driving that organises the field by input modality: image based, point cloud based, and multimodal fusion §3. It collects the sensors, datasets and metrics used in the field §2 and compares reported results on KITTI, nuScenes and Waymo §4.1. In a case study of fifteen detectors on KITTI validation, it finds 3D location error to be the largest error source and voxel-based detectors the most practical for real-time use §4.2, §3.2. All numbers are author-reported or re-run by the authors.

Problem

Given sensor input X (LiDAR signals or monocular images), a detector must output a set of oriented 3D boxes, each with a score. The authors write this as a maximum-likelihood choice of detector parameters Θ §2.1 eq. 1. Three difficulties are named:

• Images: depth is missing, and recovering it from images is an ill-posed problem §3.1.
• Point clouds: points are sparse, irregular and unordered, so a standard convolution distorts their representation §3.2.
• Fusion: the two modalities are heterogeneous (camera view versus 3D view), so finding point-to-pixel correspondences is hard §3.3.

The survey also raises safety and cost: LiDAR is expensive to deploy, and a camera-only system is not enough on its own in poor light or rain §2.2.

Method

This is a survey, so there is no single method to implement. Its contribution is a taxonomy and a set of comparisons. The taxonomy splits methods by what the detector reads at inference time §3 fig. 5 tab. 5.

Box representation. Each predicted box is parameterised by seven values for an oriented 3D box, which the survey adopts as the most common choice §2.1 fig. 4.

Image based (§3.1), in two subdivisions:

1. Result lifting §3.1.1. The problem is split into 2D detection plus depth or geometry, and 2D results are lifted to 3D with priors and constraints:
   • Mono3D scores proposals with location, shape and semantic priors, assuming objects sit near the ground plane, by minimising an energy function §3.1.1.
   • Deep3DBox uses the fact that the projected 3D box should touch the sides of its 2D box §3.1.1.
   • GS3D relies on the top centre of the 3D box projecting near the top midpoint of the 2D box §3.1.1.
   • Stereo R-CNN uses keypoints, yaw and shape on left and right proposal pairs §3.1.1.
2. Feature lifting §3.1.2. Image features are lifted into 3D:
   • Pseudo-LiDAR: estimate a depth map, back-project pixels to a point cloud, and run a point cloud detector §3.1.2. Pseudo-LiDAR++ uses sparse real LiDAR points as landmarks to correct depth bias; Pseudo-LiDAR E2E makes the pipeline trainable end to end §3.1.2.
   • Latent depth distribution: DSGN and CaDDN learn a depth distribution and build a 3D feature volume, and LIGA-Stereo uses LiDAR-derived supervision through a teacher network §3.1.2.

Point cloud based (§3.2), in three subdivisions:

1. Voxel based §3.2. Points are voxelised and processed by convolution, often collapsed to a bird's-eye-view map. VoxelNet uses dense 3D convolutions; SECOND uses sparse 3D convolutions over non-empty voxels; PointPillars uses vertical columns (pillars) in place of 3D convolutions. Later variants add Voxel R-CNN's voxel features for proposal refinement, SE-SSD's teacher-student distillation, multi-scale voxelisation, and transformer attention (CT3D, VoTr) §3.2.
2. Point based §3.2. Permutation-invariant set operators work on raw points: PointRCNN generates proposals bottom-up from foreground points; 3DSSD uses a sampling step that accounts for feature and Euclidean distance; Point-GNN reasons over a neighbourhood graph §3.2.
3. Point-voxel based §3.2. These combine both: STD, PV-RCNN, SA-SSD and BADet pair a voxel branch for efficiency with a point branch for fine detail §3.2.

Multimodal fusion (§3.3). Sequential fusion runs one modality after the other: Frustum PointNets proposes 2D boxes and then processes point frustums; Frustum-ConvNet slides frustums along the optical axis and arranges them as a BEV map. The survey argues that sequential fusion is poorly suited to safety, because a failed 2D detector breaks all later stages §3.3.

Parallel fusion keeps two independent branches and fuses them at one of three points:

• Early fusion: PointPainting appends per-point segmentation scores from the image to the raw points §3.3.
• Deep fusion: MV3D and AVOD fuse intermediate features per region of interest, ContFuse fuses multi-scale features pixel-wise, MMF adds auxiliary subtasks, and 3D-CVF uses attention to weight the two modalities §3.3.
• Late fusion: CLOCs fuses the 2D and 3D detection outputs at the decision level, using the consistency of geometry and semantics §3.3.

The survey's summary is that image based methods trail point cloud methods on KITTI, that voxel based methods are the most promising for real-time use, and that multimodal fusion still lags behind point cloud methods §3.1, §3.2, §3.3.

Key equations. Detector objective eq. 1; interpolated AP over N recall levels eq. 2 eq. 3; nuScenes detection score §2.4:

\Theta_{MLE} = \arg\max_{\Theta} \mathcal{F}\left(\{\mathbf{y}_1,\dots,\mathbf{y}_n\} \mid \mathcal{X}, \Theta\right) eq. 1

AP|_{R_N} = \frac{1}{N} \sum_{r \in R} P_{interpolate}(r) eq. 2

P_{interpolate}(r) = \max_{\tilde{r} \ge r} P(\tilde{r}) eq. 3

NDS = \frac{1}{10}\left[ 5\,mAP + \sum_{err \in \mathcal{E}} \left(1 - \min(1, err)\right) \right] §2.4

Here \mathbf{y}_i = (\mathcal{B}_i, s_i) is a box with its score, and \mathcal{E} holds the mean errors for translation, scale, orientation, attribute and velocity §2.1 §2.4.

Pseudocode: interpolated AP (how the KITTI, Waymo and nuScenes-style metrics are built from a score-sorted list; not in the paper as code) eq. 2 eq. 3 §2.4.

# Interpolated AP over N recall levels  [eq. 2] [§2.4]
# Input:  precision, recall: one entry per prediction, sorted by score
#         recall_levels: N values; KITTI uses 1/40 ... 1 since 2019 [§2.4]
# Output: AP in [0, 1]

def interpolated_ap(precision, recall, recall_levels):
    # For each level r, take the best precision at recall >= r  [eq. 3]
    interpolated = []
    for level in recall_levels:
        # Case with no point at recall >= r is not stated; 0 is assumed
        candidates = [p for p, rc in zip(precision, recall) if rc >= level]
        interpolated.append(max(candidates, default=0.0))
    # Mean over the N levels  [eq. 2]
    return sum(interpolated) / len(recall_levels)

Pseudocode: LiDAR thinning for the robustness test (the authors' procedure, reconstructed from the paper and its code; the paper cites Pseudo-LiDAR++ for it) §4.2.4 kitti_sparsify.py.

# Thin a 64-beam scan to H beams, one point per cell  [§4.2.4]
# Input:  points (N, 4): x, y, z, intensity; H in {32, 16, 8}
# Output: points (M, 4), at most one point per (row, column) cell

def sparsify_scan(points, H, W=512):
    # Keep the forward box only  [code: code/kitti_sparsify.py#L82-L88]
    points = crop_forward_box(points)
    # Vertical cell size grows as H shrinks  [code: code/kitti_sparsify.py#L48]
    row_step = radians(0.4 * 64 / H)
    # Horizontal cell size: 90 degrees over W columns  [code: ...#L49]
    col_step = radians(90 / W)
    for point in points:
        # Planar range r and full range d  [code: ...#L53-L54]
        planar, distance = ranges(point)
        # Column from azimuth  [code: ...#L57-L60]
        column = clip((radians(45) - asin(y / planar)) / col_step, 0, W - 1)
        # Row from elevation, offset by 2 degrees  [code: ...#L62-L65]
        row = clip((radians(2) - asin(z / distance)) / row_step, 0, H - 1)
        # Later points overwrite earlier ones  [code: ...#L70-L71]
        cell[row, column] = point
    # Keep every row (script default slice = 1)  [code: ...#L72]
    return non_empty_cells(cell)

Hyperparameters

This survey trains nothing. The table lists the protocol parameters the survey fixes, and those of its case study. Detector hyperparameters are not reproduced here.

Results

Author-reported. The survey's own experiments are the case study (KITTI validation, Car, moderate, AP3D R11). The KITTI, nuScenes and Waymo tables collect published results.

Headline KITTI test results (Car, AP3D R40, official server) tab. 6. The table also lists many other methods; a selection is shown.

nuScenes test tab. 7 (author-reported mAP and NDS, not per-class):

Waymo validation tab. 8 (3D mAP/mAPH, LEVEL_1 and LEVEL_2):

Case study, runtime (fifteen detectors, one GTX 1080 Ti; "-" means not reported) tab. 9:

Case study, error analysis (AP3D R11 with some box parameters replaced by ground truth) tab. 10:

Case study, robustness (AP3D R11 when the LiDAR scan is thinned to fewer beams; "n.a." as in the paper) tab. 11:

Claims stated in the text (author-reported): image based methods trail point cloud methods on KITTI because of depth ambiguity §4.1; 3D location error is the largest error contributor, followed by depth and size §4.2.3; CIA-SSD and SE-SSD lose a lot of speed under the unified protocol §4.2.2; several detectors (Point-GNN, SECOND, 3DSSD, Part-A2, SA-SSD, CIA-SSD) keep reasonable accuracy at 32 beams, and all models show an obvious drop from 32 to 16 §4.2.4.

Paper vs code

The survey's case study is the only part with code. The authors link a repository, rui-qian/SoTA-3D-Object-Detection (commit efc6744), from the paper §4.2.1. It has no license file, so nothing from it is quoted here. The code is cited only for the robustness procedure and the FLOPS counter. The README points the error-analysis archive to Hugging Face. That archive is not in the clone and was not read.

• Paper: the 64-to-N beam thinning is done "with algorithms proposed in Pseudo-LiDAR++" §4.2.4. Code: the script's header cites a different source, the Pseudo_Lidar_V2 project kitti_sparsify.py. This is an attribution difference only; the Method pseudocode follows the script.
• Paper: the thinning uses 64, 32, 16 and 8 beams §4.2.4. Code: the default is 64 rows and 512 columns, and the row step scales with 64/H kitti_sparsify.py. The paper does not state the column count.
• Paper: no crop range is stated. Code: points are kept only for 0 to 120 m forward, plus or minus 50 m lateral, and -2.5 to 1.5 m height kitti_sparsify.py.
• Paper: no rule for two points in one cell is stated. Code: the last point written wins kitti_sparsify.py. This is the behaviour of the array assignment, not a documented guarantee.
• Code: the depth-range argument --D is parsed but not passed on to the thinning function kitti_sparsify.py. It has no effect on the output.
• Code: the FLOPS counter is a modified copy of an MIT-licensed tool, and its input resolution is a parameter, not a fixed value flops_counter.py. The paper does not state which input size was used for the FLOPS column.
• No discrepancies were found for the detector hyperparameters, because the survey reuses the authors' pretrained weights and does not restate them.

Pitfalls

• Survey, not method. The card cannot be implemented directly. Use it for the taxonomy, the metric definitions and the protocol parameters, and go to the original papers for the methods.
• Author-reported and mixed sources. Most numbers come from the original papers, measured on different hardware and with different settings. The survey's own runtime comparison is the only controlled one, and the paper itself says that comparing the collected numbers "inevitably leads to controversy" §4.2.2.
• Two KITTI metrics. The case study uses 11-point interpolated AP on the validation split. The KITTI test table uses 40-point AP on the test server. Do not put the two in the same column. The protocol changed in October 2019 §2.4.
• Metric detail not stated. The AP formula does not say what precision to use when no detection reaches a recall level. The card's pseudocode assumes 0, and its comment marks this as not stated §2.4.
• Mismatch in the text. Section 4.2.3 says the largest gain from ground-truth location is 13.19%. The table gives 13.01 points for CaDDN (32.20 minus 19.19), which is the largest gain in the table. The unit (points or percent) is also unclear tab. 10 §4.2.3.
• Ordering claim. The captions of Tables 9 to 11 say the models are sorted ascending by baseline AP. Two pairs are out of order: PointRCNN (77.38) comes after SECOND (78.62), and SA-SSD (79.80) before CIA-SSD (79.74) tab. 9.
• Dataset table typo. The KITTI training count is 7,418 in the dataset table, but 7,481 in the text, which matches the official split tab. 2 §2.3.
• Error analysis is underspecified. "Replace the predicted box partly with ground truth" is defined only through the overlap threshold (0.6). The paper does not define "overlap ratio" (BEV or 3D, IoU or intersection over prediction), and it does not define "depth" as a box parameter §4.2.3.
• Robustness is a single procedure. Thinning uses one sparsification routine. Other ways of thinning a scan are not compared, and the paper's conclusion that most models hold up to 32 beams depends on it §4.2.4.
• Runtime protocol. "Unified" timing uses batch size 1 and four processes, but the paper does not state the input size or whether pre-processing is counted. Point-GNN has no unified number tab. 9 §4.2.2.
• Licensing note. The LaTeX source carries an Elsevier copyright line ("© 2022 Elsevier Ltd."). The card follows the arXiv license recorded by fetch (CC BY 4.0), as the rules require. A human reviewer should check this against the journal version before the page is marked reviewed. No figures are reproduced.

Lineage

• Builds on: none. The survey covers methods listed in its taxonomy tab. 5 and compares them on the datasets in tab. 2.
• Compared against: earlier surveys, which the authors say do not cover recent datasets or fusion variants §1.
• Superseded by: none known from content in this wiki.