V2V4Real: A Real-world Large-scale Dataset for Vehicle-to-Vehicle Cooperative Perception

In plain words

A self-driving car can miss objects hidden by occlusion or beyond its range, but a second car nearby may see them. This paper releases a dataset in which two real cars drove side by side for several days, each with a laser scanner and cameras. Every object in the scans was labelled with a 3D box, and the routes were mapped. The authors also set up three tasks on the data: finding cars in the combined scans, following them over time, and adapting a model trained on simulated driving to the real data.

Most earlier data for this problem came from simulators, and the one real dataset covered only car-to-infrastructure traffic at intersections. The paper's benchmarks show that sharing sensor information between the two cars improves detection over a single car. The benchmarks use only the laser data, so the camera images and the maps are not yet used in any task here.

TL;DR

V2V4Real is a real-world dataset from two instrumented cars driving together in Columbus, Ohio: 410 km, 20K LiDAR frames, 40K RGB frames, 240K annotated 3D boxes for 5 vehicle classes, and HD maps §3 tab. 1. It defines three benchmarks on LiDAR data: cooperative 3D detection, cooperative tracking, and Sim2Real domain adaptation from the simulated OPV2V data §4 §5. Author-reported, the best intermediate-fusion model (CoBEVT) reaches 66.5 AP@0.5 in the sync setting, against 39.8 for a single car tab. 3. Use the paper for the dataset layout and benchmark protocol. Its dataset terms are not open; see Pitfalls.

Problem

Single-vehicle perception is limited by occlusion and short range §1. Sharing sensor data between vehicles (V2V) could help, but the authors say no real-world V2V dataset existed for training and testing such methods §1. Simulated datasets (OPV2V, V2X-Sim, V2XSet) have a domain gap to real sensors and traffic §1. The only real cooperative dataset, DAIR-V2X, covers vehicle-to-infrastructure traffic, only intersections, and about 20 km of routes §1 §2. Without a real V2V dataset, the authors argue, models trained on simulation may not generalize to real driving §1.

Method

The paper contributes a data collection and annotation process, benchmark task definitions, and baseline results. It has no new network. The notes below describe what the benchmarks specify.

Data collection §3.1.

1. Two connected automated vehicles drive together in Columbus, Ohio. Their distance is kept within 150 m so that their views overlap §3.1.
2. Three days of driving cover 347 km of highway and 63 km of city road §3.1 fig. 3.
3. The authors report 19 hours and 310K raw frames. They choose 67 scenarios of 10 to 20 s each, sample at 10 Hz, and obtain 20K LiDAR frames and 40K RGB frames §3.1.
4. The two vehicles' sensor clocks are less than 50 ms apart per scene §3.1.
5. Sensors are listed in tab. 2 fig. 2.

Annotation §3.2.

1. Each vehicle's LiDAR data is labelled separately with 7-degree-of-freedom boxes (centre x, y, z; size l, w, h; yaw), using the SusTechPoint tool, by two annotator groups §3.2.
2. Five classes: car, van, pickup truck, semi-truck and bus. Each box also has a driving state (dynamic or parked) §3.2.
3. Object ids are assigned per vehicle: 0 to 1000 for the Tesla, 1001 to 2000 for the Ford. Objects seen by both vehicles are matched by BEV IoU above a threshold (value not stated), and their box sizes are unified §3.2.
4. HD maps: LiDAR frames are aggregated into a point cloud after removing dynamic objects. Consecutive frames are aligned by scan matching. GPS/IMU data and a Kalman filter correct the drift. The map is then drawn in RoadRunner, exported as OpenDRIVE and converted to lanelets §3.2.

Benchmark data protocol §4 app. C.

1. Ego vehicle: one of the two vehicles is the ego. During training the ego is picked at random. During testing the Tesla is fixed as ego §4.1.
2. Ground truth: the other vehicle's boxes are transformed into the ego frame. If the two annotations of one object differ, the one annotated in the ego frame is kept §4.1.
3. Split: 14,210 training, 2,000 validation and 3,986 test frames, shared by all three tasks §5.1.

Cooperative 3D object detection §4.1 fig. 7 app. C.1

Four fusion strategies are compared, plus a no-fusion baseline §4.1.

• No fusion: the ego uses only its own LiDAR.
• Late fusion: each vehicle detects on its own data and sends boxes and scores. The receiver applies non-maximum suppression.
• Early fusion: raw point clouds are sent and aggregated in the ego frame.
• Intermediate fusion: each vehicle encodes its LiDAR into features in the ego frame. The features are compressed with an autoencoder by 32× and fused by one of five models: F-Cooper, AttFuse, V2VNet, V2X-ViT or CoBEVT §4.1 app. C.1.

The backbone is PointPillars. The head applies two 1×1 convolutions: one for box regression (x, y, z, w, l, h, θ) on predefined anchors and one for the object-or-background score app. C.1. Only one class is evaluated: the paper merges all vehicle types into "vehicle" §4.1.

Evaluation: AP at IoU 0.5 and 0.7 within x in [-100, 100] m and y in [-40, 40] m of the ego. Transmitted size is reported as "AM" in MB. Two settings are used: Sync (no transmission delay) and Async (100 ms delay, simulated by using the non-ego LiDAR frame from the previous timestamp) §4.1.

Object tracking §4.2 app. C.2

Tracking-by-detection with the AB3DMOT tracker §5.1 §4.2. Detections come from the cooperative detector. The per-frame update is:

# Cooperative tracking, one frame  [app. C.2] [§4.2]
# Input:  detections (N_t boxes), tracks (M_t-1 trajectories)
# Output: updated tracks and their ids

def track_frame(detections, tracks):
    # Predict each track one frame ahead at constant velocity  [app. C.2]
    predicted = [predict_track(track) for track in tracks]
    # Affinity: 3D IoU between each prediction and each detection  [app. C.2]
    affinity = iou_3d(predicted, detections)    # (M, N)
    # Pair tracks and detections with the Hungarian algorithm  [app. C.2]
    pairs = hungarian(affinity)
    for track_index, detection_index in pairs:
        # Kalman update: weighted average of prediction and detection
        # [app. C.2]
        update_track(tracks[track_index], detections[detection_index])
    # New tracks start from detections that are matched in the next few
    # frames, so single false positives do not start tracks  [app. C.2]
    # Tracks are removed only after several frames without a match  [app. C.2]
    return tracks


def predict_track(track):
    # Position moves by the stored velocity: x_pred = x_(t-1) + v_x  [app. C.2]
    # Size, heading and score are copied from frame t-1  [app. C.2]
    return moved_copy(track)

Metrics are MOTA, MT, ML, AMOTA, AMOTP and sAMOTA §4.2. The paper says AMOTA and AMOTP average MOTA and MOTP over recall thresholds, and that sAMOTA spreads the score more linearly over [0, 1] §4.2.

Sim2Real domain adaptation §4.3 app. C.3

The source is the labelled simulated OPV2V dataset. The target is the unlabelled V2V4Real training LiDAR. Evaluation uses the labelled V2V4Real test set in the Sync setting §4.3. The baseline trains on OPV2V and tests on V2V4Real without adaptation. The adapted method is a domain-adversarial variant in the style of Chen et al. §4.3:

• a feature-level discriminator on the fused features. It has two 3×3 convolutions; the second maps to one channel app. C.3;
• an object-level discriminator on the classification score map. It has three linear layers with ReLU app. C.3;
• both use binary cross-entropy and a gradient reversal layer app. C.3 §4.3.

Benchmark training §5.1

Models are trained with AdamW and cosine learning-rate annealing. Early stopping chooses the best epoch. Augmentation is scaling, rotation and flip §5.1 app. D.

Hyperparameters

Sensor setup tab. 2: 2 cameras (RGB; Tesla 1280×720, Ford 2064×1544); 1 LiDAR (32 channels, 1.2M points/s, 200 m range, −25° to 15° vertical FOV, ±3 cm error, 10 Hz); GPS/IMU (Tesla RT3000, Ford Novatel SPAN E1). Both vehicles use a Velodyne VLP-32 §3.1.

Results

Author-reported. Setup: the 3,986-frame test split, PointPillars backbone, LiDAR input, and the baselines in the table. Full tables are reproduced here.

Cooperative 3D object detection, AP@IoU 0.5/0.7 tab. 3

Cooperative tracking, percentages tab. 4

Sim2Real domain adaptation, AP@IoU 0.5 tab. 5 (AP drop is measured against the same model trained on V2V4Real tab. 3; see Pitfalls)

Ablation: no data augmentation, AP@IoU 0.5 tab. 6 (value, with drop from the augmented model in brackets; detail appendix D)

Headline claims in the text, all author-reported: every cooperative method beats no fusion by at least 15.2 points overall at AP@0.5; CoBEVT is best among intermediate methods on overall AP@0.5 (1.6 above V2X-ViT in sync) §5.2. In tracking, CoBEVT has the best AMOTA, AMOTP and sAMOTA, and the text reports these as percentage-point gaps over no fusion §5.3. Domain adaptation lifts CoBEVT from 32.6 to 40.2 AP@0.5, above the no-fusion model trained on real data at 39.8 §5.4 tab. 3 tab. 5.

Paper vs code

Not compared. The official repository is ucla-mobility/V2V4Real, linked from the project page (commit recorded in the frontmatter) [project page]. Its LICENSE is the UCLA Academic Software License. It permits use only for academic or nonprofit research and forbids transfer, so it does not allow commercial reuse [code license]. Under the licensing rules for this site, no code, configuration or script from the repository is cited or quoted, and the claims above come from the paper alone.

Pitfalls

• The dataset is not CC-licensed. The paper's CC BY 4.0 license covers the paper only. The project page carries the footer "All Rights Reserved" and states no open license for data or code [project page]. Do not treat the dataset as reusable on the basis of this card.
• Only LiDAR is used in any benchmark. The HD maps and RGB images are released, but the three tasks use only point clouds (the DA task uses only LiDAR from the target domain) §4.3. The conclusion defers map and camera benchmarks to a later version §6.
• One vehicle class is evaluated. The dataset has 5 classes, but detection merges all vehicle types into one class §4.1. Per-class numbers are not reported in the main tables.
• The async setting is one frame at 10 Hz. The 100 ms delay is simulated by reusing the non-ego frame from the previous timestamp §4.1. At the 10 Hz sampling rate §3.1, that is one frame of lag. This is an inference from the two statements, not a claim the paper makes.
• Labels are one-sided. At test time the Tesla is the fixed ego. Its annotation wins where the two vehicles' boxes for one object disagree §4.1. The matching IoU threshold for shared objects is not stated §3.2.
• Tracking details are missing. The Hungarian cost (plain IoU or 1 minus IoU), any IoU gate, the Kalman noise values, and the counts for "the next few frames" (birth) and "the next couple of frames" (death) are not stated app. C.2. The "previous 3 frames plus current" input §5.1 is described without saying whether the detector or the tracker uses it.
• Domain-adaptation loss weights are missing. The weight of the discriminator loss against the detection loss, the gradient reversal scalar, and the schedule are not stated app. C.3 §4.3.
• Average DA gain differs slightly from the table. The text says domain adaptation alleviates the drop "by an average of 7.46%" §5.4. The gains in tab. 5 (+0.9, +13.7, +3.1, +12.1, +7.6) average 7.48. The unrounded values may explain the gap.
• The AP-drop column is derived. For each model, the drop in tab. 5 equals its Sync overall AP@0.5 in tab. 3 minus the value in tab. 5. This checks out for all five models, but the paper does not state the relation.
• Identical AM for intermediate fusion. All five intermediate methods report 0.20 MB tab. 3. The paper defines AM as "the transmitted data size specified by the algorithm" §4.1 and says the autoencoder compresses by 32× §4.1. The per-method size is not computed in the text.
• Fusion baselines follow original code, with one change. The authors say they mostly reuse the original configurations. The exception is V2X-ViT, whose two vehicles share one type app. C.1. The paper does not say which configurations were reused or what differs from the originals.
• Citation mismatch. Early fusion is cited to chen2019cooper in the related work §2, while F-Cooper is cited as chen2019f there and in tab. 3. Appendix C.1 cites F-Cooper as chen2019cooper app. C.1. Check the bibliography before citing the method name.
• Sim2Real needs the OPV2V source data. The paper does not state the OPV2V version or split used for training §4.3.

Lineage

• Compared against: DAIR-V2X, the earlier real-world cooperative dataset, which covers only car-to-infrastructure traffic §1 tab. 1. Simulated datasets OPV2V, V2X-Sim and V2XSet §1 tab. 1.
• Benchmarked methods: F-Cooper, AttFuse (OPV2V), V2VNet, V2X-ViT and CoBEVT §4.1 §2.
• Tracking baseline: AB3DMOT §4.2.
• Built on: PointPillars for the backbone §5.1 app. C.1; the domain-adversarial setup follows Chen et al. §4.3.
• Superseded by: not stated in this paper.