Multi-modal Sensor Fusion for Auto Driving Perception: A Survey

In plain words

A self-driving car usually has cameras and a laser scanner (LiDAR). Each sensor misses things the other sees: cameras struggle in the dark, in fog and at depth, and LiDAR returns get sparse far away and in bad weather. Combining the two is the usual remedy, and many research papers have tried it.

This survey collects those papers and asks one question about each: at which point in the pipeline are the two sensors combined? The answer gives a classification. Some methods merge raw sensor data early, some merge internal features, some merge the final detections, and some let one sensor only point the other sensor at a region. The survey also explains how the sensor data is turned into inputs for neural networks, lists the public datasets people use, and tabulates how the methods score on the KITTI benchmark. It ends with open problems: sensors that do not line up exactly, simple ways of combining features, and the lack of use of time and context.

The survey runs no experiments. Its value is the map it draws, not a method you can run.

TL;DR

• The paper is a literature review of LiDAR-camera fusion for driving perception, aimed at 2D and 3D object detection and semantic segmentation §1 §2.1.
• Its taxonomy has two major classes, strong fusion and weak fusion. Strong fusion splits into early, deep, late and asymmetry fusion, by the stage at which each branch enters the fusion §1 §4.1.
• It compares methods on the KITTI test set for bird's-eye-view (BEV) and 3D car detection tab. 2 tab. 3. These numbers are the survey's transcription, not new measurements.
• It has no code and no algorithm of its own. Treat it as a reading guide, not as a specification to implement.

Problem

Perception from a single sensor has known limits. Camera data is captured mainly low in the front view and objects can be occluded in complex scenes. LiDAR resolution varies with distance and it is affected by fog and heavy rain §1. Combining the two sensors is expected to improve perception §1.

The paper's own complaint is about the taxonomy in earlier reviews. The traditional three classes (early, deep, late) do not define the feature representation at each level. They also treat the two branches as symmetric, which hides cases where proposal-level LiDAR features are fused with data-level camera features §1.

Method

This is a survey, so "method" here means the classification scheme and the material it covers.

Task definition. Object detection draws boxes around instances of predefined categories. A 2D box is written as (x, y, h, w, c) and a 3D box as (x, y, z, h, w, l, θ, c) §2.1. Semantic segmentation assigns each pixel or 3D point one of k labels from a fixed set, or a probability for each label §2.1. Free-space detection and lane detection are described as segmentation variants §2.1.

Datasets. More than ten datasets are related to driving perception, but the survey treats three as commonly used: KITTI, Waymo and nuScenes §2.2. The summary below is taken from tab. 1 and the text of §2.2.

Split sizes: KITTI has 7481 training and 7518 test images §2.2. Waymo has 798 training, 202 validation and 150 test scenes, each 20 s long §2.2. nuScenes has 1000 scenes, split 700, 150 and 150 §2.2.

Input representations.

• Image: the usual RGB channels, or other hand-designed channels such as grey §3.1. Stereo or multi-frame cameras add depth and motion cues. Gated or infrared cameras are used in extreme conditions §3.1.
• Point-based LiDAR: each return is (x, y, z, r), where r is reflectance. Such points can be fed to point-based backbones, but the survey notes redundancy and speed costs §3.2.
• Voxel-based LiDAR: 3D space is cut into voxels. Each voxel carries a centroid and local statistics such as point density, local offset, linearity and curvature §3.3. Cylindrical voxelisation is mentioned as a more natural alternative §3.3.
• 2D-mapped LiDAR: the points are projected into a camera-plane map (CPM), which shares the image's format and can be concatenated as a channel, or into a bird's-eye view (BEV), where objects keep their length and width §3.4. The CPM is sparse because LiDAR is lower resolution than the image §3.4.

Taxonomy (the survey's main contribution). §1 §4 fig. 2

The survey says that strong fusion is "highly dependent on the LiDAR point cloud" §4.1. Weak fusion is defined by rule-based use of one modality as a supervision or selection signal for the other §4.2. Methods that combine several schemes are grouped under "other fusion methods" and are called redundant from the design view §4.3.

Classification rule. The survey gives no formal decision procedure. The table above is the reading the text supports. Several methods are placed by judgement, and the authors say some works "cannot be easily classified" into the three traditional classes §4. The sketch below is our restatement of the table as one decision list; the paper does not contain it, and the order of the tests is ours.

# Restatement of the taxonomy as a decision list (not in the paper)
# Input:  method (fusion stage of each branch)
# Output: class label

def classify(method):
    # Weak: one branch only selects or guides the other's input  [§4.2]
    if method.rule_based_guidance_only:
        return "weak-fusion"
    # Asymmetry: one branch at object level, the other below  [§4.1]
    if method.one_branch_object_level_only:
        return "asymmetry-fusion"
    # Late: both branches at object (proposal) level  [§4.1]
    if method.both_branches_object_level:
        return "late-fusion"
    # Deep: LiDAR enters at feature level  [§4.1]
    if method.lidar_feature_level:
        return "deep-fusion"
    # Early: LiDAR enters at data level  [§4.1]
    if method.lidar_data_level:
        return "early-fusion"
    # Methods that combine several schemes  [§4.3]
    return "other-fusion"

Hyperparameters

Not applicable. The survey trains and tunes no model. Its tables give results only, with no training settings tab. 2 tab. 3.

Results

Author-reported, and not re-measured. The tables summarise KITTI test-set results for the methods they list. The paper does not say whether each number was copied from the source paper or re-run. Setup: KITTI test set, car class, moderate difficulty. The survey does not state which AP variant, IoU thresholds or leaderboard snapshot it used §2.2 tab. 2 tab. 3.

Car, moderate difficulty. "-" means the method has no entry in that table.

Read off the tables: the highest 3D car-moderate value is CLOCs, a late-fusion method, at 82.28 tab. 3. The highest BEV car-moderate value is 3D-CVF, a deep-fusion method, at 89.56 tab. 2. The tables list 22 methods for BEV and 24 for 3D. The text says over 50 papers are analysed §1, but the paper does not say how the tables relate to that set.

Paper vs code

No official code. The paper links no repository or project page, and it is a literature review with no method code of its own. A GitHub repository search for the title keywords with the authors' surnames returned no repositories. The paper's tables cite many methods, each of which has its own code or none. Those are not covered by this card.

Pitfalls

• Taxonomy count disagrees across the paper. The introduction and contribution list say "two major classes, four minor classes" §1. Section 4 opens with three traditional paradigms §4. The conclusion says "three classes" §6. Use the four-plus-weak scheme from §1 and §4.1; the conclusion's wording is a leftover.
• Figure label mismatch. The asymmetry-fusion paragraph in §4.1 says "One example of late-fusion" and then points to the asymmetry figure fig. 7. The word "late" is probably a slip; the figure is the asymmetry example.
• KITTI annotation count disagrees. The text says KITTI has "more than 200k 3D object annotations" §2.2. tab. 1 lists 80k 3D boxes. The two figures are not reconciled. Use the table only if you need a count.
• Years differ between tables. F-PointNet is dated 2017 in tab. 2 and 2018 in tab. 3. SparsePool is dated 2017 in tab. 2 and 2018 in tab. 3.
• Coverage differs between tables. SCANet and PointFusion appear in tab. 3 but not tab. 2. Their BEV values are missing, not zero.
• Early fusion is defined loosely. The strict data-level definition would only allow raw RGB or grey, so the survey counts image-level segmentation results as feature-level input and so as early fusion §4.1. Methods in the early class therefore depend on that looser definition.
• Class membership is a judgement. There is no formal rule. Edge cases (for instance a method that combines early and deep fusion) are placed in "other fusion methods" §4.3.
• Metric details are missing. The table captions do not say which AP variant or IoU threshold was used tab. 2 tab. 3. Do not compare these numbers with other papers' AP values without checking the source.
• Results are not re-measured. Treat every number as the survey's transcription. Check the original paper before you quote a result.
• Implementing a method from this card is not possible. The card gives the taxonomy and the reported numbers, not architectures or training recipes. For a specific method, read its own card or its source paper.
• Open problems are stated as directions, not results. Misalignment, information loss, fusion operations (concatenation and element-wise multiply versus bilinear mapping), temporal and multi-frame information, self-supervised cross-modal learning, domain bias and resolution mismatch §5.1 §5.2 §5.3. The survey gives no experiment for any of them.

Lineage

• Builds on: the traditional three-way taxonomy (early, deep, late) and two earlier reviews, which the paper cites as the baseline it revises §1 §4.
• Compared against: the KITTI tables above tab. 2 tab. 3.
• Superseded by: not stated.