Collaborative Perception in Autonomous Driving: Methods, Datasets and Challenges

In plain words

Self-driving cars see with their own sensors, and those sensors are blocked by other vehicles, miss far-away objects, and get noisy. Collaborative perception lets nearby cars, and roadside units, share what they sense so that each car sees more than it could alone.

This paper is a survey. It does not propose a new system. It explains the three usual ways to share data (share raw sensor data, share intermediate features, or share finished detections), groups the research papers by the problem they solve, and lists the large datasets used to test these ideas. Its closing section lists open problems such as bandwidth, complex scenes, federated learning and labels.

Reading it gives a map of the field, not a recipe to build a system.

TL;DR

A survey of collaborative perception for autonomous driving, covering methods, datasets and open challenges. Use it to find which fusion scheme, communication rule or robustness fix a paper uses, and which datasets exist. Its benchmark tables are copied from original papers and are not re-run by the authors.

Problem

Individual perception suffers from occlusion, limited range at distance, and sensor noise §I. Collaboration helps, but wireless bandwidth is limited, and most methods assume perfect poses and no delay §I §III-B. The survey asks how collaboration modules are designed for ideal conditions, and how the practical issues of localization error, latency, interruption, model or task discrepancy, privacy and attacks are handled §I §III-B.

Method

The survey has no method of its own. Its structure is:

1. Collaboration schemes, by the stage at which data is shared §II fig. 3:
   • Early (data-level) collaboration fuses raw sensor data at the network input. It gives the most information and needs the most bandwidth §II.
   • Intermediate (feature-level) collaboration shares deep features and fuses them inside the network. The survey calls it the most popular choice §II.
   • Late (output-level) collaboration shares detections and merges them after post-processing. It is the cheapest in bandwidth but usually performs worst §II.
2. Methods for ideal scenarios §III-A fig. 4 fig. 5 tab. I:
   • Raw data fusion: Cooper and Coop3D §III-A.
   • Communication: greedy sharing versus agent selection (Who2com, When2com) and feature selection (FPV-RCNN, Where2comm) §III-A.
   • Feature fusion, in three families: traditional (F-Cooper, CoFF, CoCa3D); graph-based (V2VNet, DiscoNet, MP-Pose); attention-based (AttFusion, COOPERNAUT, V2X-ViT, CoBEVT, VIMI, HM-ViT) §III-A.
   • Losses: redundancy minimisation (CRCNet) and uncertainty estimation (Double-M) §III-A.
   • Output fusion with non-maximum suppression §III-A.
3. Methods for real-world issues §III-B fig. 6 tab. II:
   • Localisation error: RobustV2VNet, FPV-RCNN, OptiMatch, CoAlign §III-B.
   • Communication: latency (TCLF, V2X-ViT, SyncNet), interruption (V2X-INCOP), packet loss (LCRN) §III-B.
   • Model and task discrepancy: PSA, MPDA, STAR (TaskAgnostic) §III-B.
   • Privacy and security: P-CNN, SecPCV, P2OD, AOMAC §III-B.
4. Datasets and benchmark results §IV-A tab. III and tasks: 3D detection, 3D tracking and BEV segmentation §IV-B.
5. Challenges: transmission efficiency, complex scenes, federated learning and low label dependence §V.

The three schemes can be read as one pipeline. This sketch is illustrative and is not from the paper; it only encodes the definitions in §II fig. 3:

# Collaborative perception, one frame  [§II] [fig. 3]
# Input:  ego sensor data; data from other agents
# Output: detections for the ego vehicle

def perceive_frame(ego, others, scheme):
    # Early: send raw data, fuse before the network  [§II]
    if scheme == "early":
        cloud = transform_all(ego.raw, [o.raw for o in others])
        return detector(cloud)
    # Intermediate: send deep features, fuse inside the network  [§II]
    if scheme == "intermediate":
        features = [encoder(ego.raw)] + [o.features for o in others]
        return detector(fuse(features))
    # Late: send detections, merge after post-processing  [§II]
    detections = [detector(ego.raw)] + [o.detections for o in others]
    return postprocess(transform_all(detections))

No equations, algorithms or numbered formulas are given in the paper §III.

Hyperparameters

This is a survey. It reports no training settings for the methods it reviews.

Results

Author-reported. The survey collected these numbers from the original papers and official websites, not from its own runs §IV-B. Settings differ between the source papers, so the rows are not directly comparable across tables.

Headline numbers from the detection tables. All AP values are as printed.

Other headline numbers:

• BEV segmentation on OPV2V, vehicle IoU: CoBEVT 60.4, FuseBEVT 59.0, individual 37.7 tab. XI. Table XI lists image inputs.
• BEV segmentation on V2X-Sim, mIoU: early 42.29, late 38.38, individual 36.64, V2VNet 41.11, DiscoNet 40.84 tab. XII.
• 3D tracking on V2V4Real, AMOTA: CoBEVT 32.1, late 29.3, individual 16.1 tab. IX.
• 3D tracking on V2X-Sim, MOTA: DiscoNet 56.7, V2VNet 55.3, early 58.0, late 21.5 tab. X.

Paper vs code

No official code. The paper's project page is a GitHub repository, CatOneTwo/Collaborative-Perception-in-Autonomous-Driving, that holds the paper's reference list and summary tables. It is not implementation code, so no code field is set. The project page was opened with WebFetch, which found markdown files and figures only and no code.

Pitfalls

• No runnable method. Nothing in the paper can be implemented as written. Use the cited original papers for any method, and check the code they release.
• Tables are copied, not re-run. Results come from different papers with different backbones, settings and splits. Do not rank methods across rows of different tables. The survey says so for the different real-world settings §IV-B.
• Table and figure numbers are inferred. The numbers above follow the order of \caption commands in the LaTeX source. The compiled PDF was not checked, so verify a number there before citing it outside this card.
• Internal inconsistencies to check against the original sources:
  • The text says late collaboration "always" performs worst §II. The tables show Who2com and When2com below late collaboration on V2X-Sim detection tab. IV and segmentation tab. XII, and the text itself notes this §IV-B.
  • The V2X-Sim segmentation text says "six categories" §IV-B. tab. XII lists seven classes plus mIoU.
  • The V2X-Sim segmentation table lists identical rows for Who2com and When2com tab. XII. Verify both against the source paper before relying on either.
  • The OPV2V segmentation table cites V2VNet to OPV2V tab. XI, not to V2VNet.
  • The V2X-Sim scene text says 2 to 5 vehicles per scene §IV-A. tab. III says 1 to 5 agents.
• Late and early below individual under noise. On the noisy V2XSet setting, late (30.7) and early (38.4) collaboration score below individual perception (40.2) tab. VI. The survey notes that pose and delay problems are the reason robustness methods are needed §III-B; it does not analyse this table further.
• Budget. Early collaboration needs high bandwidth and is hard to run in real time §II. Intermediate methods reduce bandwidth but depend on feature selection that the survey says may drop low-confidence areas §V.
• Simulation vs real data. Most datasets are simulated; only DAIR-V2X and V2V4Real are real §IV-A. Privacy methods were evaluated on individual-perception data, not on collaborative datasets §III-B.

Lineage

• Compares itself with earlier reviews by Caillot et al., Ren et al. and Cui et al. (2022), and claims that it covers the latest datasets and frameworks §I.
• Builds on: no earlier method. It cites the methods above as the sources of its tables.
• Superseded by: not known from content in this wiki.