OPV2V: An Open Benchmark Dataset and Fusion Pipeline for Perception with Vehicle-to-Vehicle Communication

In plain words

Self-driving cars see the road with their own sensors, and a car often cannot see a vehicle that is hidden behind a truck or another car. If nearby cars could share what they see, each car would get a fuller picture. The problem is that there was no large, public dataset for testing this kind of car-to-car sharing, so researchers could not compare their methods fairly.

This paper builds such a dataset in a driving simulator, with several cars driving together and recording their laser scans and camera images. It then tests several ways of combining the cars' data, from sharing raw scans to sharing only final detections. The proposed method shares compact summaries of each car's view, and lets each car weigh how useful every other car's summary is before combining them. The authors report that this keeps detection quality high even when the summaries are squeezed to be much smaller, which matters because the radio link between cars has limited bandwidth.

TL;DR

OPV2V is a simulated open dataset for vehicle-to-vehicle (V2V) perception with LiDAR, collected in CARLA and a digital copy of Culver City §3.1. The authors benchmark four LiDAR detectors under four fusion strategies, 16 models in all §5.1 tab. 4. Their Attentive Intermediate Fusion pipeline shares compressed intermediate features and fuses them with self-attention across vehicles §4. The result is author-reported, and the card's implementation details come from the paper text plus the released code.

Problem

Single-vehicle detection degrades when objects are heavily occluded or small, and a single car's LiDAR returns are too sparse to recover them §1. Sharing information between connected vehicles gives several viewpoints of the same object §1. Before this work, V2V methods were either trained on adapted single-vehicle datasets, which the authors argue does not reflect V2V geometry, or on data that was never released §2. Without a shared benchmark, methods could not be compared §1 §2.

Sharing raw data gives the best information but needs very high bandwidth; sharing detection outputs needs little bandwidth but depends on each car's own detector §2. Intermediate features are the middle ground the authors target §2 §4.

Method

Dataset generation §3.1

1. Simulate cooperative driving with OpenCDA, a co-simulation tool that runs on CARLA and SUMO, in eight default CARLA towns §3.1.
2. Give each connected automated vehicle (CAV) four cameras, one 64-channel LiDAR and GPS/IMU sensors. Sensor data is streamed at 20 Hz and recorded at 10 Hz §3.1 tab. 1.
3. Build a digital town of Culver City from its road topology, selected buildings and simulated traffic, and record four scenes of about 600 frames that the authors use for domain-gap validation §3.1.
4. Export LiDAR, RGB images and bird's-eye-view (BEV) maps for each CAV per frame. The authors report 11,464 frames, 232,913 boxes and 249.4 GB in total §3.1 §3.2 tab. 2 tab. 3.

Benchmark §5.1

5. Run four LiDAR detectors (SECOND, VoxelNet, PIXOR, PointPillar) under four settings: no fusion (ego only), late fusion, early fusion and intermediate fusion.
6. Early fusion projects every CAV's point cloud into the ego frame using the shared poses and runs one detector on the union §5.1.
7. Late fusion has each CAV detect on its own and sends its boxes and scores to the ego, which applies NMS §5.1.

Attentive Intermediate Fusion §4

The pipeline runs in six stages, from exchanging poses to predicting boxes: each car exchanges its pose, extracts features, compresses them, sends them, then the ego attends across all received features and predicts §4 fig. 7. Each CAV broadcasts its relative pose and extrinsics, which defines a graph of CAVs within communication range §4. The ego is chosen per scenario §4. Each neighbour extracts features from its own projected point cloud with the detector's backbone §4. The steps below follow the paper; the attention details come from code.

# Attentive intermediate fusion for one ego vehicle  [§4] [fig. 7]
# Input:  ego features (C, H, W); neighbours as (features, distance_m)
# Output: fused ego features (C, H, W)

def attentive_fusion(ego_features, neighbours):
    # 1. Keep only neighbours inside the 70 m communication range  [§5.2]
    in_range = [f for f, d in neighbours if d <= 70.0]
    # 2. Each neighbour encodes its features to a smaller bottleneck
    #    [§4]
    codes = [encode(f) for f in in_range]  # (C / 2^k, H / 2^k, W / 2^k)
    # 3. The ego decodes the received codes back to full size  [§4]
    decoded = [decode(c) for c in codes]  # (C, H, W) each
    # 4. Stack the ego and its neighbours, one row per vehicle  [§4]
    stacked = stack([ego_features] + decoded)  # (N, C, H, W)
    # 5. Attend across vehicles at each (h, w) location  [§4]
    fused = vehicle_self_attention(stacked)  # (N, C, H, W)
    # 6. Keep only the ego's fused output for the prediction head  [§4]
    return fused[0]  # (C, H, W)

The attention step, as implemented in the released code, is:

# Self-attention across vehicles at each spatial location  [§4]
# Input:  stacked features (N, C, H, W)
# Output: attended features (N, C, H, W)

def vehicle_self_attention(stacked):
    # 1. Turn each spatial location into a sequence of N vehicles  [§4]
    tokens = reshape(stacked, (H * W, N, C))  # (HW, N, C)
    # 2. Dot products between vehicles, scaled by sqrt(C), with no
    #    learned projections
    #    [code: opencood/models/fuse_modules/self_attn.py#L37-L40]
    scores = tokens @ transpose(tokens) / sqrt(C)  # (HW, N, N)
    # 3. Softmax over the vehicles
    #    [code: opencood/models/fuse_modules/self_attn.py#L41]
    weights = softmax(scores, axis=-1)  # (HW, N, N)
    # 4. Weighted sum of the vehicles' features
    #    [code: opencood/models/fuse_modules/self_attn.py#L42]
    mixed = weights @ tokens  # (HW, N, C)
    return reshape(mixed, (N, C, H, W))

Where the fusion module sits in each detector:

• PIXOR: three fusion modules at the 2D backbone outputs, at 192, 256 and 384 channels §5.1 pixor_intermediate.py.
• PointPillar, SECOND and VoxelNet: fusion at each backbone level [code: opencood/models/sub_modules/att_bev_backbone.py#L62] [code: opencood/models/sub_modules/att_bev_backbone.py#L124-L127]. The paper says the four detectors add the module "in a similar way" §5.1.

Hyperparameters

Values in the paper text come with citations. Values marked "code default" are from the released configs, which the paper does not state.

Results

Author-reported. Setup: OPV2V test split, CARLA default towns and the Culver City digital town, AP at IoU 0.5 and 0.7 §5.2 tab. 4. The paper's bold highlighting is not reproduced here.

tab. 4

Findings the paper states:

• Every fusion method gains at least 10% AP at IoU 0.7 over no fusion, on both test sets §5.4.
• Early fusion beats late fusion, which the authors attribute to keeping more sensing detail §5.4.
• Intermediate fusion is best everywhere except PIXOR on Culver City, where early fusion is best §5.4 tab. 4.
• The Culver City results are generally lower. The authors attribute this to a domain gap and to dense traffic at rush hour §5.4.
• AP rises with the number of CAVs, but the gain slows at four vehicles. The paper gives this curve only as a figure (VoxelNet, IoU 0.7) §5.5 fig. 9.
• With VoxelNet at a 4096× compression rate, AP drops by about 3% and still beats early and late fusion. The paper gives the rate-versus-AP curve as a figure §5.6 fig. 10.
• At 27 Mbps and 300 m, the authors estimate about 5 ms to send the message at 4096× compression §5.6.

Paper vs code

Code: raw/code/2109.07644 at commit 31ba16025da27ffe4e336f011290dfbc66f9a1f1, the default-branch HEAD at clone time. It is the authors' OpenCOOD repository, linked from the paper's footnote §3.1. Its LICENSE is the UCLA Academic Software License, so the repo may not be used commercially. We cite it only.

• Compression encoder. The paper describes 2D convolutions with max pooling §4. The code uses stride-2 3×3 convolutions with batch norm and ReLU and no max pooling auto_encoder.py. The decoder uses 2×2 transposed convolutions, as the paper says [code: opencood/models/sub_modules/auto_encoder.py#L37-L46].
• Where the bottleneck is. The paper says the bottleneck is broadcast and the decoder runs on the ego §4. The code applies encoder and decoder in one forward pass on the same device, so no transmission is modelled [code: opencood/models/voxel_net_intermediate.py#L168-L173]. The 5 ms figure is computed separately from data size and a fixed 27 Mbps rate [code: opencood/data_utils/datasets/basedataset.py#L104] [code: opencood/data_utils/datasets/basedataset.py#L367].
• Attention projections. The paper says "self-attention models are adopted" §4 and does not describe query, key or value projections. The code uses the features directly as query, key and value, with no learned projections self_attn.py. The attention runs over vehicles at each location [code: opencood/models/fuse_modules/self_attn.py#L56-L58].
• Output. The code keeps only the ego's row of the attended output [code: opencood/models/fuse_modules/self_attn.py#L59-L61]. This matches the paper's ego-centric fusion §4.
• Ego selection. The paper says the ego is random during training and fixed at test time §4 footnote. The dataset class takes the agent with the minimum id as ego basedataset.py, and the files read here show no random choice in training.
• Early stopping. The paper says the models are trained with early stop §5.3. The training loop saves checkpoints every save_freq epochs and evaluates every eval_freq epochs train.py. No early-stop criterion was found in the files read.
• Evaluation range. The paper gives x ∈ [−140, 140] m §5.2. The PointPillar config uses ±140.8 m [code: opencood/hypes_yaml/point_pillar_intermediate_fusion.yaml#L26].
• Communication range. Both the paper §5.2 and the code use 70 m [code: opencood/data_utils/datasets/init.py#L20] [code: opencood/data_utils/datasets/intermediate_fusion_dataset.py#L97-L103].
• Compression default. The default PointPillar and VoxelNet configs set compression to 0, which disables it [code: opencood/hypes_yaml/point_pillar_intermediate_fusion.yaml#L79] [code: opencood/hypes_yaml/voxelnet_intermediate_fusion.yaml#L66]. The paper's compressed results need a non-zero setting that the paper does not state.

Pitfalls

• Learning rate, epochs, batch size and schedule are not in the paper. The code defaults in the Hyperparameters table are the only concrete values. They may not be what produced tab. 4.
• The compression rate is undefined in the paper. The code halves channels and halves each spatial side per layer [code: opencood/models/sub_modules/auto_encoder.py#L24-L33]. If the rate counts tensor elements, each layer divides them by 8, so 4096× would be four layers. This is our inference from the code, not a stated fact. Check it against the compiled paper before relying on it.
• Two copies of Table 4 exist in the LaTeX source. A commented-out older table gives different numbers, for example VoxelNet intermediate fusion at default AP@0.5/0.7 of 0.905/0.804 vs 0.906/0.864 in the published table [src: root.tex lines 388–408]. Use the published Table 4. The compiled PDF was not checked.
• Figures and tables are cited from LaTeX order. Figure and table numbers follow the order of the LaTeX environments. Check them against the PDF.
• Ego choice matters for evaluation. The paper fixes the ego at test time for fairness §4 footnote. A different ego changes the scores, so keep the same rule when reproducing.
• Dataset scale. Data is 249.4 GB, and the Culver City subset is about 600 frames §3.1 tab. 2 tab. 3. Expect large storage and long simulation times.
• Hidden cost. Training took 14 days on four RTX 3090 GPUs §5.3. The attention runs at every spatial location with N × N weights, so cost grows with the number of vehicles, though the paper does not measure this.
• Dataset license. The paper text read here does not state a license for the dataset; it only says the dataset will be released §1. The code is non-commercial. Do not use the dataset commercially until its license is known.
• License notices disagree within the code. self_attn.py says "TDG-Attribution-NonCommercial-NoDistrib" [code: opencood/models/fuse_modules/self_attn.py#L3], but the repo LICENSE is the UCLA Academic Software License. Both are non-commercial.

Lineage

• Builds on: intermediate feature sharing for V2V perception, as in V2VNet and F-Cooper §2. Named here, not linked: no card exists for them.
• Compared against: early and late fusion baselines, the four LiDAR detectors SECOND, VoxelNet, PIXOR and PointPillar §5.1.
• Uses: the 70 m communication range from V2VNet §5.2.
• Superseded by: not stated in this card's sources.