V2X-Seq: A Large-Scale Sequential Dataset for Vehicle-Infrastructure Cooperative Perception and Forecasting

In plain words

A self-driving car sees only what its own sensors can reach. Sensors mounted on the traffic infrastructure, such as a pole at a junction, can see around corners and behind other vehicles. The paper asks how to use that roadside view to follow the objects on the road over time and to predict where they will go next.

Research in this area lacked real recordings of this kind, with video-like sequences from both the car and the roadside. The authors built a dataset from intersections in one Chinese city. It holds sequences of camera and laser-scanner frames with tracked 3D boxes, plus road maps and traffic-light signals. Two parts are released: one for tracking objects in 3D, and one with many short trajectories for forecasting.

The paper also defines three tasks on this data and gives baseline results. The main proposed method, FF-Tracking, avoids sending full sensor data from the road to the car. It sends compact features plus a per-feature rate of change, and the car estimates where the features will be when its own data arrives. This keeps the message small and makes the result less sensitive to delay. The numbers are the authors' own.

TL;DR

V2X-Seq releases real vehicle and infrastructure sequences for 3D tracking (15k+ frames, 95 scenes) and trajectory forecasting (210k scenarios from 672 h at 28 intersections), with three benchmark tasks. FF-Tracking transmits a compressed infrastructure feature and its flow, then predicts the feature at the ego timestamp with a linear step. Author-reported MOTA at 200 ms latency is 52.26 for FF-Tracking against 51.27 for early fusion. Infrastructure trajectories lower TNT's minADE from 12.01 to 8.27 on the cooperative-view validation set. The tracking training code is not in the released repository, so the tracking results cannot be reproduced from it. The forecasting code exists but its defaults differ from the paper.

Problem

Single-vehicle perception is limited by its field of view, and single-vehicle forecasting lacks interaction context §1 §3.2. Existing public driving datasets are mostly single-vehicle, and the earlier DAIR-V2X-C release targets frame-by-frame 3D detection without trajectories §2 tab. 1. Cooperative tracking and forecasting need synchronised sequences from both sides of the road, with stable object identities §3.1 §4.

For a tracking method, two costs compete. Sending raw sensor data uses the most bandwidth, sending only detections loses information, and sending intermediate features is the paper's chosen middle ground §4. Latency between the roadside frame and the ego frame also causes fusion errors, because objects move in the meantime §4.

Method

The paper contributes three parts: a dataset (Section 3), three task definitions with metrics (Sections 4 and 5), and one method, FF-Tracking, for the 3D tracking task §1.

Dataset construction

• Sequential perception dataset (SPD). 95 scenes built on DAIR-V2X-C, each 10 to 20 s long, sampled at 10 Hz, with vehicle and infrastructure images and point clouds §3.1. Ten object classes carry 3D boxes (7 values: x, y, z, width, length, height, yaw), occlusion and truncation attributes, and each object has a persistent tracking ID §3.1. Cooperative tracking labels were made by converting the roadside boxes into the ego frame, matching them with the Hungarian method on Euclidean distance, scoring the matches by trajectory similarity, dropping low-scoring matches and refining the rest by hand §3.1 fig. 2.
• Trajectory forecasting dataset (TFD). Scenarios come in three kinds: about 50,000 where roadside and vehicle data are fused, and about 80,000 each from the roadside alone and from the vehicle alone. Each holds 10 s of tracked objects at 10 Hz, a local vector map and, for the cooperative and infrastructure views, real-time traffic-light signals §3.2 fig. 3. Object types come from 8 classes. Cooperative trajectories were generated like the cooperative tracking labels but without manual refinement §3.2.
• Vector maps and traffic lights. Lane centrelines, crosswalks and stop lines as line segments, with turn attributes and lane width. Coordinates carry a constant offset. Traffic-light records (time, location, colour, shape, remaining time) are taken at 10 Hz from the signals themselves §3.3.
• Scope of data. The source tree holds an appendix with sensor layout and a trajectory-mining procedure (10 s segments with 5 s overlap, scoring, filtering), but the main file does not include it §3.2 (source only; see Pitfalls).

Task 1: VIC3D Tracking

Inputs are the ego vehicle's sequential frames and pose, plus the infrastructure's sequential frames and pose up to an earlier time, because communication delays the roadside data §4. Outputs are the category, location, orientation and tracking ID of objects around the ego vehicle. The ground truth is the union of the ego and infrastructure labels, restricted to the ego region of interest §4 eq. 1.

GT = (GT_{v} \cup GT_{i}) \cap R eq. 1

Metrics are MOTA, MOTP and ID switches (IDS), plus BPS, the bytes per second sent from infrastructure to vehicle §4.

FF-Tracking

FF-Tracking is a middle-fusion pipeline built on FFNet's feature-flow prediction §4. It transmits a static infrastructure feature F_0 and a feature flow F_1, not a single feature. The feature at the ego timestamp is predicted with a linear step, where t is the time offset §4 eq. 2:

F_{future}(t) = F_0 + t \cdot F_1 eq. 2

The steps, in the order the paper lists them §4 fig. 4:

# FF-Tracking, one roadside frame to ego tracks  [§4] [fig. 4]
# Input: infrastructure frames up to t_i; ego point cloud at t_v
# Output: 3D boxes with persistent tracking IDs

def ff_tracking(infra_frames, ego_points, time_offset):
    # 1. Extract feature and flow from past infrastructure frames  [§4]
    feature, flow = extract_feature_and_flow(infra_frames)
    # 2. Compress, send and decompress both tensors  [§4]
    feature, flow = decompress(compress(feature, flow))
    # 3. Predict the feature at the ego timestamp  [§4] [eq. 2]
    infra_feature = feature + time_offset * flow
    # 4. Move it into the ego frame and fuse with the ego feature  [§4]
    ego_feature = extract_ego_feature(ego_points)
    fused = fuse(to_ego_frame(infra_feature), ego_feature)
    # 5. Detect with SSD, then track with AB3DMOT  [§4]
    boxes = ssd_detect(fused)
    return ab3dmot_track(boxes)

The paper defers to FFNet for the flow-prediction configuration §4. The compressor and decompressor layouts, the feature extractor and the flow estimator are not described in this paper's main text.

Task 2: Online-VIC forecasting

Given the target agent's past positions (50 frames, 5 s), the roadside trajectories and traffic-light signals in the scene, the local vector map, and the ego vehicle's past trajectory, predict the target's future coordinates for the next 50 frames (5 s) §5.1. Latency is set to zero here, so the roadside and ego times are the same §5.1. Evaluation uses Top-K predictions with K = 6 and the metrics minADE, minFDE and missing rate (MR) §5.1.

Task 3: Offline-VIC forecasting

The model is pre-trained on infrastructure trajectories and then fine-tuned for the vehicle view. At test time it uses only ego-vehicle data, with the same 50-frame observation, 50-frame horizon and K = 6 §5.2.

Forecasting baselines

• Baseline 1 (no infrastructure data). TNT and HiVT trained on ego trajectories and vector maps from the cooperative-view scenes, for 30 epochs §6.2.1.
• Baseline 2 (PP-VIC, online). Infrastructure and ego trajectories are first fused with CBMOT, keeping only infrastructure tracks that are complete or have high detection scores. TNT or HiVT then predicts on the fused tracks, 30 epochs §6.2.1.
• Baseline 3 (offline). TNT or HiVT pre-trained on the 80,000 infrastructure-view trajectories, then fine-tuned on cooperative-view ego data §6.2.1.

Only trajectories and vector maps within 50 m of the ego vehicle are encoded §6.2.1.

Hyperparameters

Results

Author-reported. Setup: the SPD test split for tracking (four classes, the region above) and the TFD validation part of the 50,000 cooperative-view scenes for forecasting §6.1 §6.2. Full tables reproduced from the source.

Tracking (SPD), tab. 2:

Source: tab. 2. The authors bold early fusion at 0 ms (56.03) and FF-Tracking at 200 ms (52.26) as best results. The paper's text says only MOTA is compared tab. 2 §6.1.2.

Forecasting (TFD validation), tab. 3, K = 6:

Source: tab. 3. Text reports the minADE reductions as 3.74 (TNT online), 0.28 (HiVT online), 7.65 (TNT offline) and 0.03 (HiVT offline) §6.2.2, which match the table.

Dataset scale, tab. 1 and text: 15,000+ perception frames from 95 scenes; 210,000 forecasting scenarios from 28 intersections; 672 h of data §1 §3. The paper's comparison table lists 583 h for the forecasting part and "0..43" h for the perception part, which does not sum to the 672 h stated in the abstract (see Pitfalls).

Paper vs code

The repository is AIR-THU/DAIR-V2X-Seq (commit 8a5b4f82…), linked from the abstract. Its owner is the authors' institute organisation, and the paper names it, so it is treated as official. It has no LICENSE file, so no verbatim code is quoted here.

• Tracking training code is not released. The repository's TODO list leaves "Training & Evaluation code for VIC3D Tracking Benchmark" unchecked README.md. The FF-Tracking training and the early, late and V2VNet baselines therefore cannot be checked against code. The README's SPD benchmark link points to a different repository path, AIR-THU/DAIR-V2X, which returned HTTP 404 when fetched, so it was not verified README.md.
• FFNet link is not FF-Tracking. The repository links FFNET (haibao-yu/FFNet-VIC3D) as related work README.md. A fetched summary of that page describes a 3D detection code base, not the tracking method or the flow equation, so it is not used as code for this card.
• Forecasting: reported numbers differ from the paper. The repository's benchmark table gives TNT without infrastructure 8.45 / 17.93 / 0.77, online 8.00 / 16.65 / 0.75, offline 4.38 / 9.37 / 0.62; HiVT without infrastructure 1.44 / 2.52 / 0.36, online 1.26 / 2.34 / 0.35, offline 1.39 / 2.23 / 0.32 README.md. Paper tab. 3 gives TNT 12.01 / 24.15 / 0.84 (none), 8.27 / 17.25 / 0.76 (online), 4.36 / 9.23 / 0.62 (offline), and HiVT 1.55 / 2.59 / 0.36, 1.27 / 2.36 / 0.35, 1.52 / 2.27 / 0.30. The repository's TNT-without-infrastructure row is far lower than the paper's, so the two were probably produced by different runs or versions. The paper gives no explanation.
• Epoch counts. The paper trains the forecasting baselines for 30 epochs §6.2.1. The HiVT trainer defaults to 64 epochs [code: projects/HiVT_plugin/train.py#L25]. The TNT trainer defaults to 50 epochs train.py. The fine-tuning scripts set 10 epochs hivt_finetune.sh tnt_finetune.sh. The pre-training and training scripts pass no epoch count, so the defaults apply.
• HiVT optimiser. Code defaults are AdamW with lr 3e-4, weight decay 1e-4, cosine annealing with T_max 64, embedding width 64, dropout 0.1 and 6 modes hivt.py hivt.py. The paper says only "the other settings remain the same as the original" §6.2.1.
• TNT optimiser. Code defaults are batch 128, lr 1e-3, weight decay 0.01, and lr multiplied by 0.9 every 5 epochs [code: projects/TNT_plugin/train.py#L99-L127]. The trainer class defaults decay to 0.3 tnt_trainer.py, so the effective value depends on which entry point runs. The paper states none of these.
• Metrics. ADE is the mean L2 error over the horizon and FDE the final-step L2 error ade.py fde.py. The MR threshold is 2.0 m mr.py, which the paper does not state. Metrics are computed on the selected best prediction per agent hivt.py.
• Local radius and data. The 50 m local radius matches the code default tfd_datamodule.py hivt.py. The code takes index 49 as the current frame of a 50-frame history [code: projects/HiVT_plugin/datasets/tfd_dataset.py#L147].
• Seed. The HiVT scripts fix seed 2022 [code: projects/HiVT_plugin/train.py#L13]; the paper does not mention seeds.

No other discrepancies were checked. The code for the PP-VIC CBMOT fusion was not read.

Pitfalls

• FF-Tracking cannot be rebuilt from this card or the repository. The compressor, feature extractor, flow estimator, SSD and AB3DMOT settings are deferred to FFNet, and the tracking training code is absent README.md. Treat the tracking numbers as author-reported only.
• Time unit in eq. 2. The paper does not say how t is measured (seconds, frames or a fraction of the interval), nor which time reference F_0 and F_1 use §4 eq. 2. An implementer must choose this and check it against the latency in the evaluation.
• BPS text and table disagree. Section 6.1.2 cites 6.2×10⁵ and 1.2×10⁶ Byte/s for middle fusion at 0 ms. In tab. 2 the 0 ms FF-Tracking row is 6.2×10⁵ and the 1.2×10⁶ value is the 200 ms row §6.1.2 tab. 2. The cost comparison at 0 ms is therefore unclear.
• Bandwidth gap. Early fusion sends about 1.3×10⁷ Byte/s, roughly 20 times the middle-fusion rate of 6.2×10⁵ Byte/s, while also scoring higher at 0 ms (56.03 vs 54.75 MOTA) tab. 2 §6.1.2. The advantage of middle fusion depends on the bandwidth budget the reader assumes.
• Latency claims beyond the table. The claim that FF-Tracking beats early fusion by up to 4 MOTA at 300 ms rests on fig. 5 only; no table value is given §6.1.2.
• Hour totals. tab. 1 lists "0..43" h for the perception part and 583 h for forecasting, against 672 h in the abstract and in §1. The arithmetic is inconsistent, and the table value "0..43" looks like a typo.
• Latency ignored in forecasting. The forecasting tasks set the roadside time equal to the ego time, so the latency problem is not tested there §5.1.
• Coordinates are offset. Map and trajectory coordinates carry a constant offset added for data security §3.3. Absolute positions are not physically meaningful, but relative geometry is preserved.
• Class scope. SPD annotates ten classes, but tracking evaluates only Car, Van, Bus and Truck inside the fixed rectangle §3.1 §6.1. Pedestrians and cyclists are excluded from the reported tracking scores.
• Cooperative labels contain manual refinement. SPD cooperative tracking labels were refined by hand, but TFD cooperative trajectories were not §3.1 §3.2. The labels therefore differ in quality between the two parts.
• Dropped matches. Low-score cooperative matches are dropped in both the tracking and the TFD labels, so the cooperative sets are smaller than the union of the single-view sets §3.1 §3.2. The amount dropped is not stated.
• Baseline 2 thresholds. The filter that keeps only complete or high-score infrastructure tracks has no stated threshold §6.2.1.
• Baseline 3 pre-training. Epochs and learning rate for pre-training on the 80,000 infrastructure trajectories are not stated §6.2.1.
• Typo in text. Section 6.2.2 writes "minPDE"; the metric is minFDE §6.2.2.
• Commented-out table in source. The source contains an older tracking table inside comments with different BPS values (for example 2.2×10³ and 1.7×10³ where the live table has 1.3×10⁷ and 6.2×10⁵). It is not part of the paper. Use only the live tab. 2.
• Appendix not in main file. The source tree has a supplementary file with sensor layouts and the trajectory-mining details (10 s segments, 5 s overlap, scoring). The main file does not input it, so whether these details appear in the published PDF is not verified. Cite them only after checking the PDF.
• Source-only figure. fig. 4 and fig. 5 carry the method overview and the latency curves; their values cannot be read from the text beyond what is stated in §6.1.2.

Lineage

• Builds on: FFNet, which introduced feature-flow prediction for vehicle infrastructure cooperation §4; DAIR-V2X-C, which supplies the SPD scenes §3.1; TNT and HiVT as forecasting baselines §6.2.1; V2VNet, PointPillars, AB3DMOT, SSD, CBMOT and the Hungarian method as components §4 §6.1.1.
• Compared against: early, middle and late fusion for tracking §6.1.1; TNT and HiVT with and without infrastructure data for forecasting §6.2.1.
• Superseded by: not stated in this wiki.