Collaborative perception

Overview

A self-driving car often cannot see an object that is hidden behind another vehicle or a truck 2109.07644 §1. Connected vehicles can share what they sense, which gives several viewpoints of the same object 2109.07644 §1. Every variant follows the same pattern: each vehicle or roadside unit produces a representation of its view, the representations are sent to an ego vehicle, and the ego combines them before it predicts boxes or a path. The variants differ in what that representation is, and that choice sets the trade-off between accuracy and bandwidth. Sharing raw data keeps the most information but needs very high bandwidth. Sharing detection outputs needs little bandwidth but depends on each car's own detector. Intermediate features sit between the two 2109.07644 §2. UniV2X goes further and sends sparse object and lane queries plus one occupancy map, not feature maps 2404.00717 § Data generation.

This page rests on four papers. OPV2V introduces a simulated benchmark and an intermediate-fusion pipeline 2109.07644 §1 §4. V2X-ViT adds roadside units and a Transformer fusion module, evaluated on its own simulated dataset 2203.10638 §4.1. V2V4Real is the real-world counterpart: two instrumented cars, LiDAR benchmarks for detection and tracking, and a Sim2Real adaptation test 2303.07601 §3 §4. UniV2X takes camera images from a vehicle and a roadside unit and plans the vehicle's path end to end, evaluated on DAIR-V2X 2404.00717 § Method § Experiments: DAIR-V2X Dataset. Each paper evaluates on its own data, so the numbers below are not comparable across papers. OPV2V and V2XSet are simulated; V2V4Real and DAIR-V2X use real recordings. The disagreements are in Contradictions.

Variants

No fusion (baseline)

Each vehicle detects from its own sensors and nothing is shared 2109.07644 §5.1. In the benchmark it has the lowest AP at IoU 0.7 in every detector row of the default and Culver City test sets 2109.07644 tab. 4.

Late fusion (share detections)

Each vehicle runs its detector and sends its boxes and scores to the ego, which applies non-maximum suppression 2109.07644 §5.1. For VoxelNet on the default towns, AP@0.7 is 0.738 2109.07644 tab. 4. Card: OPV2V.

Early fusion (share raw point clouds)

Each vehicle projects its LiDAR points into the ego frame using the shared poses, and one detector runs on the combined cloud 2109.07644 §5.1. The authors attribute its advantage over late fusion to keeping more sensing detail 2109.07644 §5.4. For VoxelNet on the default towns, AP@0.7 is 0.758 2109.07644 tab. 4. Sharing raw data needs very high bandwidth 2109.07644 §2.

Intermediate fusion (share compressed features)

Each vehicle runs the detector's backbone, compresses the features through an encoder, and sends them. The ego decodes them, attends across all vehicles at each spatial location, and predicts from its own fused output 2109.07644 §4. The paper calls its pipeline Attentive Intermediate Fusion. For VoxelNet on the default towns, AP@0.7 is 0.864, the highest value in its table 2109.07644 tab. 4. With VoxelNet at a 4096× compression rate, AP drops by about 3% and still exceeds early and late fusion 2109.07644 §5.6. The paper gives the rate-versus-AP curve only as a figure, and it does not state how a rate maps to encoder layers 2109.07644 §5.6 fig. 10.

Intermediate fusion with heterogeneous attention (V2X-ViT)

Roadside units join the fusion next to the vehicles. The pipeline first warps each agent's feature map to the ego pose with a differentiable 2-D affine transform, which corrects pose error and delay 2203.10638 §3.1 app. B.1. It then adds a delay encoding to every agent's features 2203.10638 §3.2.3. Three encoder layers follow. Each layer applies attention across agents, with projections chosen by node type (vehicle or infrastructure) and by link type, then window attention at three scales, then an MLP 2203.10638 §3.2.1 §3.2.2 app. B.3. The ablation adds one module at a time, and AP@0.7 in the noisy setting rises from 0.478 to 0.614 with all of them on 2203.10638 tab. 2. The baselines in its table also receive infrastructure data, so the gain comes from the fusion module and not from the roadside input 2203.10638 §4.1 §4.2. Card: V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer.

Query-level fusion with occupancy (UniV2X)

The roadside unit sends a different payload. It sends its tracked agent queries and lane queries, each 256-dimensional, plus one occupancy probability map and a flow that predicts how that map moves 2404.00717 § Data generation. The ego advances the roadside queries to its own time with the predicted flow 2404.00717 eq. 2. It then matches its queries to the roadside queries with the Hungarian method on the distance between their reference points, fuses each matched pair, and appends the roadside queries that found no match 2404.00717 § Cross-view query matching. Occupancy maps are merged by max 2404.00717 § Cross-view occupancy fusion. The output is a planned path, not boxes 2404.00717 § Planning output. On DAIR-V2X, the authors report an average collision rate of 0.34% against 0.89% for no fusion. Their transmission figure is 8.09e5 BPS, against 8.19e7 for BEV feature fusion 2404.00717 tab. 2. Card: End-to-End Autonomous Driving through V2X Cooperation.

Evidence on real vehicles (V2V4Real)

V2V4Real adds no new fusion method. It runs the same families on real LiDAR from two cars: late fusion, early fusion, and five intermediate methods (F-Cooper, AttFuse, V2VNet, V2X-ViT and CoBEVT). The intermediate features are compressed 32× by an autoencoder, and every model uses PointPillars 2303.07601 §4.1 app. C.1. In the synchronous setting, AP@0.5 is 55.0 for late fusion, 59.7 for early fusion, 60.7 to 66.5 for the five intermediate methods, and 39.8 with no fusion 2303.07601 tab. 3. CoBEVT has the best AP@0.5 at 66.5, and V2X-ViT has the best AP@0.7 at 36.9 2303.07601 tab. 3. The authors list the transmitted size per message as 0.20 MB for each intermediate method, against 0.96 MB for early fusion and 0.003 MB for late fusion 2303.07601 tab. 3. Card: V2V4Real: A Real-world Large-scale Dataset for Vehicle-to-Vehicle Cooperative Perception.

When to use which

The OPV2V and V2XSet numbers come from simulated datasets. The V2V4Real and DAIR-V2X numbers come from real recordings, but from different sensors, sites and setups. Treat all of them as a guide to ordering, not as expected values for other sensors or sites.

Training cost is a real factor for intermediate fusion. The OPV2V authors report about 14 days on four RTX 3090 GPUs for all models 2109.07644 §5.3. UniV2X trains in four stages on eight A100 GPUs 2404.00717 app. Training § Experiments: Implementation.

Implementation sketch

The core is the same for all intermediate-fusion variants: encode, send, decode, then fuse across vehicles. The pseudocode below follows the paper's Attentive Intermediate Fusion 2109.07644 §4. The paper does not give the learned projections of its attention, so the query, key and value step is an open choice. V2X-ViT replaces the attention step with its heterogeneous encoder; its pseudocode is on its card.

# Intermediate fusion for one ego vehicle  [arxiv:2109.07644 §4]
# Input:  ego features (C, H, W); neighbours as (features, distance_m)
# Output: fused ego features (C, H, W)

def fuse(ego_features, neighbours):
    # 1. Keep neighbours inside the communication range of 70 m  [arxiv:2109.07644 §5.2]
    in_range = [f for f, d in neighbours if d <= 70.0]
    # 2. Each neighbour encodes its features into a smaller bottleneck  [arxiv:2109.07644 §4]
    codes = [encode(f) for f in in_range]
    # 3. The ego decodes the received codes back to full size  [arxiv:2109.07644 §4]
    decoded = [decode(c) for c in codes]  # (C, H, W) each
    # 4. Stack the ego and its neighbours, one row per vehicle  [arxiv:2109.07644 §4]
    stacked = stack([ego_features] + decoded)  # (N, C, H, W)
    # 5. Attend across vehicles at each spatial location  [arxiv:2109.07644 §4]
    attended = vehicle_self_attention(stacked)  # (N, C, H, W)
    # 6. Keep only the ego's row for the prediction head  [arxiv:2109.07644 §4]
    return attended[0]  # (C, H, W)

Query-level fusion (UniV2X) changes the payload and the fusion step. The roadside sends queries and a probability map, not features. The ego fuses by Hungarian matching of query reference points, not by attention across vehicles 2404.00717 § Cross-view query matching. Its released code differs from the paper in ways listed under Contradictions.

Where the paper's implementation differs from the paper text, see Contradictions.

Contradictions

Across papers.

• Early fusion versus intermediate fusion. OPV2V reports intermediate fusion above early fusion, with VoxelNet at AP@0.7 of 0.864 against 0.758 on the default towns 2109.07644 tab. 4. In V2X-ViT's table, the OPV2V row is below early fusion in the perfect setting, at 0.664 against 0.710, and above it in the noisy setting, at 0.487 against 0.384 2203.10638 tab. 1. V2V4Real's real-car table puts all five intermediate methods above early fusion at AP@0.5 in the sync setting, at 60.7 to 66.5 against 59.7. At AP@0.7, F-Cooper is below early fusion, at 31.8 against 32.1 2303.07601 tab. 3. The datasets, detectors and implementations differ, and no one of these papers settles the ordering.
• Which intermediate method is best. In V2V4Real, CoBEVT has the best AP@0.5 and V2X-ViT the best AP@0.7 2303.07601 tab. 3. In V2X-ViT's own table, V2X-ViT is first on V2XSet in every column 2203.10638 tab. 1. The winner changes with the dataset and the metric, so do not rank these methods across papers.
• Compression rates. OPV2V reports results up to 4096× and does not define its rate 2109.07644 §5.6. V2X-ViT's default rate is 32 and it tests up to 128× 2203.10638 §4.2 §4.3. Its code maps rate 32 to 8 channels from 256 naive_compress.py. V2V4Real also reports 32× with an autoencoder whose architecture is not stated 2303.07601 §4.1. The rates are not comparable.
• Datasets and baselines. UniV2X reports car-class mAP of 0.268 for V2X-ViT and 0.243 for early fusion on DAIR-V2X 2404.00717 tab. 3. V2X-ViT reports its own numbers on V2XSet 2203.10638 tab. 1. UniV2X re-implements its baselines with BEVFormer, except CoCa3D [arxiv:2404.00717 § Experiments: Agent perception]. The numbers do not measure the same thing, so do not rank V2X-ViT against UniV2X from them.
• UniV2X transmission figures. The same method has a planning figure of 8.09e5 BPS 2404.00717 tab. 2 and a tracking figure of 6.96e4 BPS 2404.00717 tab. 3. The text gives 6.94e4 for the tracking figure 2404.00717 § Experiments: Agent perception. The paper does not say what the planning figure includes.

Within OPV2V. The paper text and the authors' released code differ in these points 2109.07644 §4 2109.07644 §5.3:

• Compression encoder. The paper describes 2D convolutions with max pooling. The code uses strided convolutions and no max pooling [code: opencood/models/sub_modules/auto_encoder.py#L14-L33].
• Attention projections. The paper does not describe query, key or value projections. The code uses the features directly, with no learned projections self_attn.py.
• Ego selection. The paper says the ego is random during training and fixed at test time 2109.07644 §4. The dataset class takes the agent with the minimum id as ego basedataset.py.
• Early stopping. The paper says the models use early stopping 2109.07644 §5.3. The training loop saves and evaluates checkpoints on a fixed schedule, and no early-stop criterion was found in the files read train.py.

The paper also gives the published Table 4 with values that differ from an older table in its LaTeX source. The published table is the one used here 2109.07644 tab. 4.

Within V2X-ViT. The paper text and the released code differ in these points 2203.10638 §3.1 app. B.3:

• Compression. The paper describes 1 × 1 convolutions with a default rate of 32. The code uses a 3 × 3 convolution with batch norm and ReLU, and the released config sets the rate to 0, which disables it [code: v2xvit/models/sub_modules/naive_compress.py#L5-L24] [code: v2xvit/hypes_yaml/point_pillar_v2xvit.yaml#L84].
• Learning-rate schedule. The paper says the rate drops by ×0.1 every 10 epochs 2203.10638 §4.2. The code uses multistep decay at epochs 15 and 50 train_utils.py.

Within UniV2X. The paper text and the released code differ in these points 2404.00717 eq. 2 § Cross-view query matching:

• Latency compensation. The paper advances roadside queries with a predicted flow 2404.00717 eq. 2, and its latency result depends on it 2404.00717 tab. 9. The released fusion forward has no time-compensation step agent_fusion.py. The latency result cannot be reproduced from this code alone.
• Matching gate. The paper uses Euclidean distance with Hungarian matching. The code also rejects pairs whose offset on either axis is larger than the vehicle box agent_fusion.py.
• Occupancy threshold order. The paper applies max fusion to probabilities, then thresholds. The code thresholds each map first [code: projects/mmdet3d_plugin/univ2x/dense_heads/occ_head.py#L541-L546]. With one shared threshold the results are the same.
• BEV grid. The paper gives a 0.25 m grid over [-50, 50] m, but 200 cells over that range is 0.5 m per cell 2404.00717 § Experiments: Implementation. The code uses a 102.4 m range and a 200 × 200 BEV, about 0.512 m per cell univ2x_coop_e2e.py.

Papers

• OPV2V: the OPV2V dataset, a 16-model LiDAR benchmark across four fusion settings, and the Attentive Intermediate Fusion pipeline.
• V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer: a Transformer that fuses vehicle and roadside LiDAR features, with delay and pose correction, evaluated on the authors' simulated V2XSet.
• V2V4Real: A Real-world Large-scale Dataset for Vehicle-to-Vehicle Cooperative Perception: a real two-car LiDAR and camera dataset, with benchmarks for cooperative detection, tracking and Sim2Real adaptation.
• End-to-End Autonomous Driving through V2X Cooperation: UniV2X, a camera-based planner that fuses roadside queries and occupancy with the vehicle's own queries, evaluated on DAIR-V2X.