--- kind: concept slug: collaborative-perception title: Collaborative perception summary: >- Connected vehicles share what they sense, so each sees objects it would miss alone. Variants differ in what crosses the link: detections (late), raw point clouds (early), compressed features (intermediate) or object queries (UniV2X). Author-reported: intermediate fusion led on OPV2V and on V2V4Real at AP@0.5; under pose and delay noise on V2XSet, late and early fusion fell below no fusion. tags: [autonomous-driving, multi-agent-perception, lidar, camera, v2x] papers: ["2109.07644", "2203.10638", "2303.07601", "2404.00717"] related: [bird-eye-view-perception, self-attention, vision-transformer, multi-agent-systems, object-queries, bipartite-matching, domain-adaptation] status: draft updated_at: 2026-11-01 --- ## Overview A self-driving car often cannot see an object that is hidden behind another vehicle or a truck [arxiv:2109.07644 §1]. Connected vehicles can share what they sense, which gives several viewpoints of the same object [arxiv:2109.07644 §1]. Every variant follows the same pattern: each vehicle or roadside unit produces a representation of its view, the representations are sent to an ego vehicle, and the ego combines them before it predicts boxes or a path. The variants differ in what that representation is, and that choice sets the trade-off between accuracy and bandwidth. Sharing raw data keeps the most information but needs very high bandwidth. Sharing detection outputs needs little bandwidth but depends on each car's own detector. Intermediate features sit between the two [arxiv:2109.07644 §2]. UniV2X goes further and sends sparse object and lane queries plus one occupancy map, not feature maps [arxiv:2404.00717 § Data generation]. This page rests on four papers. OPV2V introduces a simulated benchmark and an intermediate-fusion pipeline [arxiv:2109.07644 §1] [§4]. V2X-ViT adds roadside units and a Transformer fusion module, evaluated on its own simulated dataset [arxiv:2203.10638 §4.1]. V2V4Real is the real-world counterpart: two instrumented cars, LiDAR benchmarks for detection and tracking, and a Sim2Real adaptation test [arxiv:2303.07601 §3] [§4]. UniV2X takes camera images from a vehicle and a roadside unit and plans the vehicle's path end to end, evaluated on DAIR-V2X [arxiv:2404.00717 § Method] [§ Experiments: DAIR-V2X Dataset]. Each paper evaluates on its own data, so the numbers below are not comparable across papers. OPV2V and V2XSet are simulated; V2V4Real and DAIR-V2X use real recordings. The disagreements are in Contradictions. ## Variants ### No fusion (baseline) Each vehicle detects from its own sensors and nothing is shared [arxiv:2109.07644 §5.1]. In the benchmark it has the lowest AP at IoU 0.7 in every detector row of the default and Culver City test sets [arxiv:2109.07644 tab. 4]. ### Late fusion (share detections) Each vehicle runs its detector and sends its boxes and scores to the ego, which applies non-maximum suppression [arxiv:2109.07644 §5.1]. For VoxelNet on the default towns, AP@0.7 is 0.738 [arxiv:2109.07644 tab. 4]. Card: [OPV2V](/papers/2109.07644). ### Early fusion (share raw point clouds) Each vehicle projects its LiDAR points into the ego frame using the shared poses, and one detector runs on the combined cloud [arxiv:2109.07644 §5.1]. The authors attribute its advantage over late fusion to keeping more sensing detail [arxiv:2109.07644 §5.4]. For VoxelNet on the default towns, AP@0.7 is 0.758 [arxiv:2109.07644 tab. 4]. Sharing raw data needs very high bandwidth [arxiv:2109.07644 §2]. ### Intermediate fusion (share compressed features) Each vehicle runs the detector's backbone, compresses the features through an encoder, and sends them. The ego decodes them, attends across all vehicles at each spatial location, and predicts from its own fused output [arxiv:2109.07644 §4]. The paper calls its pipeline Attentive Intermediate Fusion. For VoxelNet on the default towns, AP@0.7 is 0.864, the highest value in its table [arxiv:2109.07644 tab. 4]. With VoxelNet at a 4096× compression rate, AP drops by about 3% and still exceeds early and late fusion [arxiv:2109.07644 §5.6]. The paper gives the rate-versus-AP curve only as a figure, and it does not state how a rate maps to encoder layers [arxiv:2109.07644 §5.6] [fig. 10]. ### Intermediate fusion with heterogeneous attention (V2X-ViT) Roadside units join the fusion next to the vehicles. The pipeline first warps each agent's feature map to the ego pose with a differentiable 2-D affine transform, which corrects pose error and delay [arxiv:2203.10638 §3.1] [app. B.1]. It then adds a delay encoding to every agent's features [arxiv:2203.10638 §3.2.3]. Three encoder layers follow. Each layer applies attention across agents, with projections chosen by node type (vehicle or infrastructure) and by link type, then window attention at three scales, then an MLP [arxiv:2203.10638 §3.2.1] [§3.2.2] [app. B.3]. The ablation adds one module at a time, and AP@0.7 in the noisy setting rises from 0.478 to 0.614 with all of them on [arxiv:2203.10638 tab. 2]. The baselines in its table also receive infrastructure data, so the gain comes from the fusion module and not from the roadside input [arxiv:2203.10638 §4.1] [§4.2]. Card: [V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer](/papers/2203.10638). ### Query-level fusion with occupancy (UniV2X) The roadside unit sends a different payload. It sends its tracked agent queries and lane queries, each 256-dimensional, plus one occupancy probability map and a flow that predicts how that map moves [arxiv:2404.00717 § Data generation]. The ego advances the roadside queries to its own time with the predicted flow [arxiv:2404.00717 eq. 2]. It then matches its queries to the roadside queries with the Hungarian method on the distance between their reference points, fuses each matched pair, and appends the roadside queries that found no match [arxiv:2404.00717 § Cross-view query matching]. Occupancy maps are merged by max [arxiv:2404.00717 § Cross-view occupancy fusion]. The output is a planned path, not boxes [arxiv:2404.00717 § Planning output]. On DAIR-V2X, the authors report an average collision rate of 0.34% against 0.89% for no fusion. Their transmission figure is 8.09e5 BPS, against 8.19e7 for BEV feature fusion [arxiv:2404.00717 tab. 2]. Card: [End-to-End Autonomous Driving through V2X Cooperation](/papers/2404.00717). ### Evidence on real vehicles (V2V4Real) V2V4Real adds no new fusion method. It runs the same families on real LiDAR from two cars: late fusion, early fusion, and five intermediate methods (F-Cooper, AttFuse, V2VNet, V2X-ViT and CoBEVT). The intermediate features are compressed 32× by an autoencoder, and every model uses PointPillars [arxiv:2303.07601 §4.1] [app. C.1]. In the synchronous setting, AP@0.5 is 55.0 for late fusion, 59.7 for early fusion, 60.7 to 66.5 for the five intermediate methods, and 39.8 with no fusion [arxiv:2303.07601 tab. 3]. CoBEVT has the best AP@0.5 at 66.5, and V2X-ViT has the best AP@0.7 at 36.9 [arxiv:2303.07601 tab. 3]. The authors list the transmitted size per message as 0.20 MB for each intermediate method, against 0.96 MB for early fusion and 0.003 MB for late fusion [arxiv:2303.07601 tab. 3]. Card: [V2V4Real: A Real-world Large-scale Dataset for Vehicle-to-Vehicle Cooperative Perception](/papers/2303.07601). ## When to use which The OPV2V and V2XSet numbers come from simulated datasets. The V2V4Real and DAIR-V2X numbers come from real recordings, but from different sensors, sites and setups. Treat all of them as a guide to ordering, not as expected values for other sensors or sites. | Situation | Use | Why | |---|---|---| | No V2V link, or no fusion training data | No fusion | Nothing to share. It is the baseline the paper measures against [arxiv:2109.07644 §5.1] | | Link has enough bandwidth, clean poses and detail matters most | Early fusion | Sharing raw data keeps the most information [arxiv:2109.07644 §2] [§5.4]. With pose and delay noise it fell to 0.384 AP@0.7, below no fusion at 0.402 [arxiv:2203.10638 tab. 1]. On real cars the five intermediate methods beat it at AP@0.5, but F-Cooper is below it at AP@0.7 [arxiv:2303.07601 tab. 3] | | Link is narrow and each car's own detector is the unit you want to keep | Late fusion | Needs little bandwidth, but quality depends on each car's detector [arxiv:2109.07644 §2]. With noise it fell to 0.307 AP@0.7 [arxiv:2203.10638 tab. 1] | | Link is narrow and you can train a fusion module | Intermediate fusion (OPV2V) | Highest AP in most settings, with about 3% loss at 4096× compression [arxiv:2109.07644 tab. 4] [§5.6]. V2X-ViT's table ranks it differently; see Contradictions | | Roadside units are deployed, or pose and delay are noisy | Heterogeneous attention (V2X-ViT) | Noisy-setting AP@0.7 is 0.614, against 0.487 for the OPV2V row in its table [arxiv:2203.10638 tab. 1]. Under pose noise up to 0.2 m, it loses under 3% AP, while other intermediate methods lose at least 6% [arxiv:2203.10638 §4.3] | | Latency budget is tight | V2X-ViT with one encoder layer (V2X-ViT-S) | 28 ms against 57 ms on a Tesla V100, with noisy AP@0.7 of 0.591 against 0.614 [arxiv:2203.10638 tab. 4] | | Real sensors, and the only training data is simulated | Intermediate fusion with domain-adversarial adaptation (V2V4Real Sim2Real) | Adaptation raised CoBEVT from 32.6 to 40.2 AP@0.5 on real test data, above 39.8 for no fusion trained on real data [arxiv:2303.07601 tab. 5] [§5.4]. The source is labelled OPV2V, whose version the paper does not state [§4.3]. The loss weights and gradient-reversal scalar are not stated [app. C.3] | | Only cameras are on the car and the roadside | Query-level fusion (UniV2X) | Its experiments use image input only [arxiv:2404.00717 § Experiments: Implementation]. The LiDAR variants above do not apply | | The ego needs a planned path, and the link is narrow | Query-level fusion (UniV2X) | Planning transmission of 8.09e5 BPS, against 8.19e7 for BEV feature fusion [arxiv:2404.00717 tab. 2]. Collision rate is lower than no fusion, but average L2 is 3.48 m, not lower than no fusion at 3.44 m [arxiv:2404.00717 tab. 2] | | You need a V2V or V2X dataset to compare methods | OPV2V, V2XSet, V2V4Real or DAIR-V2X, after checking their licenses | The OPV2V paper releases its data [arxiv:2109.07644 §1]. V2V4Real is the real-car V2V dataset the authors describe [arxiv:2303.07601 §1]. None of the four cards records an open dataset license. V2V4Real's card notes that its project page reserves all rights, although the paper is CC BY 4.0 ([V2V4Real](/papers/2303.07601)). Do not use any of them commercially until a data license is found | Training cost is a real factor for intermediate fusion. The OPV2V authors report about 14 days on four RTX 3090 GPUs for all models [arxiv:2109.07644 §5.3]. UniV2X trains in four stages on eight A100 GPUs [arxiv:2404.00717 app. Training] [§ Experiments: Implementation]. ## Implementation sketch The core is the same for all intermediate-fusion variants: encode, send, decode, then fuse across vehicles. The pseudocode below follows the paper's Attentive Intermediate Fusion [arxiv:2109.07644 §4]. The paper does not give the learned projections of its attention, so the query, key and value step is an open choice. V2X-ViT replaces the attention step with its heterogeneous encoder; its pseudocode is on [its card](/papers/2203.10638). ```pseudocode # Intermediate fusion for one ego vehicle [arxiv:2109.07644 §4] # Input: ego features (C, H, W); neighbours as (features, distance_m) # Output: fused ego features (C, H, W) def fuse(ego_features, neighbours): # 1. Keep neighbours inside the communication range of 70 m [arxiv:2109.07644 §5.2] in_range = [f for f, d in neighbours if d <= 70.0] # 2. Each neighbour encodes its features into a smaller bottleneck [arxiv:2109.07644 §4] codes = [encode(f) for f in in_range] # 3. The ego decodes the received codes back to full size [arxiv:2109.07644 §4] decoded = [decode(c) for c in codes] # (C, H, W) each # 4. Stack the ego and its neighbours, one row per vehicle [arxiv:2109.07644 §4] stacked = stack([ego_features] + decoded) # (N, C, H, W) # 5. Attend across vehicles at each spatial location [arxiv:2109.07644 §4] attended = vehicle_self_attention(stacked) # (N, C, H, W) # 6. Keep only the ego's row for the prediction head [arxiv:2109.07644 §4] return attended[0] # (C, H, W) ``` Query-level fusion (UniV2X) changes the payload and the fusion step. The roadside sends queries and a probability map, not features. The ego fuses by Hungarian matching of query reference points, not by attention across vehicles [arxiv:2404.00717 § Cross-view query matching]. Its released code differs from the paper in ways listed under Contradictions. Where the paper's implementation differs from the paper text, see Contradictions. ## Contradictions **Across papers.** - **Early fusion versus intermediate fusion.** OPV2V reports intermediate fusion above early fusion, with VoxelNet at AP@0.7 of 0.864 against 0.758 on the default towns [arxiv:2109.07644 tab. 4]. In V2X-ViT's table, the OPV2V row is below early fusion in the perfect setting, at 0.664 against 0.710, and above it in the noisy setting, at 0.487 against 0.384 [arxiv:2203.10638 tab. 1]. V2V4Real's real-car table puts all five intermediate methods above early fusion at AP@0.5 in the sync setting, at 60.7 to 66.5 against 59.7. At AP@0.7, F-Cooper is below early fusion, at 31.8 against 32.1 [arxiv:2303.07601 tab. 3]. The datasets, detectors and implementations differ, and no one of these papers settles the ordering. - **Which intermediate method is best.** In V2V4Real, CoBEVT has the best AP@0.5 and V2X-ViT the best AP@0.7 [arxiv:2303.07601 tab. 3]. In V2X-ViT's own table, V2X-ViT is first on V2XSet in every column [arxiv:2203.10638 tab. 1]. The winner changes with the dataset and the metric, so do not rank these methods across papers. - **Compression rates.** OPV2V reports results up to 4096× and does not define its rate [arxiv:2109.07644 §5.6]. V2X-ViT's default rate is 32 and it tests up to 128× [arxiv:2203.10638 §4.2] [§4.3]. Its code maps rate 32 to 8 channels from 256 [code: v2xvit/models/sub_modules/naive_compress.py#L5-L24]. V2V4Real also reports 32× with an autoencoder whose architecture is not stated [arxiv:2303.07601 §4.1]. The rates are not comparable. - **Datasets and baselines.** UniV2X reports car-class mAP of 0.268 for V2X-ViT and 0.243 for early fusion on DAIR-V2X [arxiv:2404.00717 tab. 3]. V2X-ViT reports its own numbers on V2XSet [arxiv:2203.10638 tab. 1]. UniV2X re-implements its baselines with BEVFormer, except CoCa3D [arxiv:2404.00717 § Experiments: Agent perception]. The numbers do not measure the same thing, so do not rank V2X-ViT against UniV2X from them. - **UniV2X transmission figures.** The same method has a planning figure of 8.09e5 BPS [arxiv:2404.00717 tab. 2] and a tracking figure of 6.96e4 BPS [arxiv:2404.00717 tab. 3]. The text gives 6.94e4 for the tracking figure [arxiv:2404.00717 § Experiments: Agent perception]. The paper does not say what the planning figure includes. **Within OPV2V.** The paper text and the authors' released code differ in these points [arxiv:2109.07644 §4] [arxiv:2109.07644 §5.3]: - **Compression encoder.** The paper describes 2D convolutions with max pooling. The code uses strided convolutions and no max pooling [code: opencood/models/sub_modules/auto_encoder.py#L14-L33]. - **Attention projections.** The paper does not describe query, key or value projections. The code uses the features directly, with no learned projections [code: opencood/models/fuse_modules/self_attn.py#L39-L43]. - **Ego selection.** The paper says the ego is random during training and fixed at test time [arxiv:2109.07644 §4]. The dataset class takes the agent with the minimum id as ego [code: opencood/data_utils/datasets/basedataset.py#L179-L181]. - **Early stopping.** The paper says the models use early stopping [arxiv:2109.07644 §5.3]. The training loop saves and evaluates checkpoints on a fixed schedule, and no early-stop criterion was found in the files read [code: opencood/tools/train.py#L181-L185]. The paper also gives the published Table 4 with values that differ from an older table in its LaTeX source. The published table is the one used here [arxiv:2109.07644 tab. 4]. **Within V2X-ViT.** The paper text and the released code differ in these points [arxiv:2203.10638 §3.1] [app. B.3]: - **Compression.** The paper describes 1 × 1 convolutions with a default rate of 32. The code uses a 3 × 3 convolution with batch norm and ReLU, and the released config sets the rate to 0, which disables it [code: v2xvit/models/sub_modules/naive_compress.py#L5-L24] [code: v2xvit/hypes_yaml/point_pillar_v2xvit.yaml#L84]. - **Learning-rate schedule.** The paper says the rate drops by ×0.1 every 10 epochs [arxiv:2203.10638 §4.2]. The code uses multistep decay at epochs 15 and 50 [code: v2xvit/tools/train_utils.py#L199-L202]. **Within UniV2X.** The paper text and the released code differ in these points [arxiv:2404.00717 eq. 2] [§ Cross-view query matching]: - **Latency compensation.** The paper advances roadside queries with a predicted flow [arxiv:2404.00717 eq. 2], and its latency result depends on it [arxiv:2404.00717 tab. 9]. The released fusion forward has no time-compensation step [code: projects/mmdet3d_plugin/univ2x/fusion_modules/agent_fusion.py#L131-L199]. The latency result cannot be reproduced from this code alone. - **Matching gate.** The paper uses Euclidean distance with Hungarian matching. The code also rejects pairs whose offset on either axis is larger than the vehicle box [code: projects/mmdet3d_plugin/univ2x/fusion_modules/agent_fusion.py#L61-L66]. - **Occupancy threshold order.** The paper applies max fusion to probabilities, then thresholds. The code thresholds each map first [code: projects/mmdet3d_plugin/univ2x/dense_heads/occ_head.py#L541-L546]. With one shared threshold the results are the same. - **BEV grid.** The paper gives a 0.25 m grid over [-50, 50] m, but 200 cells over that range is 0.5 m per cell [arxiv:2404.00717 § Experiments: Implementation]. The code uses a 102.4 m range and a 200 × 200 BEV, about 0.512 m per cell [code: projects/configs_e2e_univ2x/univ2x_coop_e2e.py#L13-L29]. ## Papers - [OPV2V](/papers/2109.07644): the OPV2V dataset, a 16-model LiDAR benchmark across four fusion settings, and the Attentive Intermediate Fusion pipeline. - [V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer](/papers/2203.10638): a Transformer that fuses vehicle and roadside LiDAR features, with delay and pose correction, evaluated on the authors' simulated V2XSet. - [V2V4Real: A Real-world Large-scale Dataset for Vehicle-to-Vehicle Cooperative Perception](/papers/2303.07601): a real two-car LiDAR and camera dataset, with benchmarks for cooperative detection, tracking and Sim2Real adaptation. - [End-to-End Autonomous Driving through V2X Cooperation](/papers/2404.00717): UniV2X, a camera-based planner that fuses roadside queries and occupancy with the vehicle's own queries, evaluated on DAIR-V2X.