End-to-End Autonomous Driving through V2X Cooperation

In plain words

A self-driving car sees the road with cameras on its own body, which gives it a limited view. A camera on a roadside pole can see more. This paper asks how the car can use that roadside view while the whole driving system, from seeing to planning, is trained as one network, instead of tuning each stage on its own.

The roadside sends only a compact summary: a few "object tokens" for vehicles and lane pieces, plus one grid of where things are likely to be occupied and how that grid moves. The car lines up these summaries with its own tokens, merges the matched pairs, adds the roadside objects it did not see itself, and then plans a path. The message is much smaller than sending dense features.

TL;DR

UniV2X is a query-based network that takes camera images from a vehicle and from a roadside infrastructure unit, and predicts the vehicle's future waypoints end to end § Problem formulation § Method. The roadside side transmits a sparse-dense hybrid: agent queries and lane queries, plus one dense occupancy probability map and a linear flow for it § Data generation. The vehicle fuses these by spatial alignment, Hungarian matching and max fusion § Agent fusion § Occupancy fusion. The authors report better collision and off-road rates than the no-fusion baseline on DAIR-V2X, at a transmission cost far below dense BEV fusion tab. 2. All numbers are author-reported. The released code (Apache-2.0) differs from the paper in several places, listed under Paper vs code.

Problem

Single-vehicle driving with low-cost cameras has a limited view § Introduction. Most V2X work improves one module, such as detection, tracking or mapping, and does not optimise the final planning output § Introduction. Sending raw data or dense BEV features is too expensive for limited links, and sending only detections loses context § Problem formulation: Data for transmission. Three conditions are named: transmitted data must help planning, be cheap to send, and be checkable § Problem formulation: Challenges. The paper assumes communication latency (the roadside frame is older than the vehicle frame) and positioning or calibration error between the two views § Problem formulation § Ego identification and removing.

Method

The input is ego-vehicle images with the pose at the vehicle timestamp, and roadside images with the pose at an earlier roadside timestamp. The output is the ego vehicle's future coordinates § Problem formulation § Method. Each side runs BEVFormer for a BEV feature, then TrackFormer for agent queries, MapFormer for lane queries and an occupancy head for the occupancy map. The roadside transmits only the filtered agent queries, the filtered lane queries and the occupancy map with its flow § Data generation. The vehicle then does the following, in order § Cross-view agent fusion [§ Cross-view lane fusion] § Cross-view occupancy fusion § Planning output:

1. Temporal synchronisation: roadside agent queries are advanced to the vehicle time with a predicted query flow eq. 2. The paper describes this as a three-layer MLP trained with self-supervision [§ Temporal synchronization].
2. Spatial synchronisation: roadside reference points are moved into the vehicle frame with the relative pose. The rotation matrix, flattened to nine values, is concatenated to the query and passed through an MLP eq. 3.
3. Matching: vehicle and roadside queries are paired with the Hungarian method on the distance of their reference points § Cross-view query matching. Matched pairs are fused into the vehicle query; unmatched roadside queries are appended § Cross-view query matching.
4. Ego removal: roadside detections that sit on the ego vehicle are removed § Ego identification and removing.
5. Decoder augmentation: the synchronised roadside queries are cross-attended to refine the vehicle BEV feature, to produce agent intermediate outputs eq. 4 § Decoder input augmentation.
6. Lane fusion: lane queries are synchronised spatially and concatenated with the vehicle lane queries to speed up training § Cross-view lane fusion.
7. Occupancy fusion: roadside maps are expanded over future steps with the flow eq. 1, moved into the vehicle frame with the pose, and fused with the vehicle maps by max § Cross-view occupancy fusion.
8. Planning: fused agent, lane and occupancy features feed a UniAD-style motion and planning stack that produces rough waypoints; the binary occupancy map is then used to shift waypoints away from occupied grids and towards lane-compliant positions § Planning output.

Sparse-dense hybrid transmission § Data generation:

• Agent queries: 256-dimensional features with tracking ids and reference points for each valid agent; the paper uses N_a^inf for the count, which is not stated numerically.
• Lane queries: 256-dimensional, sent with their reference points, after removing low-scoring queries.
• Occupancy: the dense BEV feature (200 x 200 x 256) goes through an MLP to a 200 x 200 probability map P_inf.
• Flow: a probability-flow module gives the future map as a linear function of time eq. 1. Sending T future maps costs T x 200 x 200 floats, while sending the present map and its flow costs 2 x 200 x 200 floats [§ Data generation].

The decoder-augmentation step adds an MLP output of the synchronised roadside queries to the vehicle BEV feature, a residual update eq. 4 [§ Decoder input augmentation].

Training is in four stages app. Training: (1) pre-train the roadside and the vehicle systems separately; (2) train the two fusion modules with the tracking and mapping losses eq. 5; (3) train all fusion modules and all vehicle modules with the combined loss eq. 6; (4) train the flow networks self-supervised on roadside frame pairs with a similarity loss. The appendix says the roadside queries used in stage 3 are the stored ones from inference mode, because training-clip queries gave unstable inference results.

# Roadside side: what is sent each step  [§ Data generation]
# Inputs: roadside images; outputs are the four transmitted items
def roadside_message(images):
    # BEV feature (200, 200, 256) from a BEVFormer backbone  [§ Data generation]
    bev = bevformer(images)
    # Tracked agent queries, with ids and reference points  [§ Data generation]
    agent_queries = track_queries(bev)
    # Lane queries after removing low-scoring ones  [§ Data generation]
    lane_queries = map_queries(bev)
    # Dense occupancy probability map, shape (200, 200)  [§ Data generation]
    occ_present = occupancy_mlp(bev)
    # Probability flow P_1; the internals of this module are not stated  [eq. 1]
    occ_flow = probability_flow(bev)
    return agent_queries, lane_queries, occ_present, occ_flow

# Vehicle side: fusion of roadside agent queries  [§ Cross-view agent fusion]
# Inputs: inf_queries (N_i, 256), veh_queries (N_v, 256), relative pose
def fuse_agents(inf_queries, veh_queries, inf_points, veh_points, pose):
    # Latency compensation: advance roadside queries to the vehicle time
    # [eq. 2]; not found in the released code, see Paper vs code
    inf_queries = advance_by_flow(inf_queries, time_gap)
    # Move roadside reference points into the vehicle frame
    # [code: agent_fusion.py#L152-L156]
    inf_points = transform_points(inf_points, pose)
    # Remove the first roadside query inside the ego rectangle
    # [code: agent_fusion.py#L158-L176]
    inf_queries, inf_points = remove_ego_query(inf_queries, inf_points)
    # Only vehicle queries with score >= 0.05 may be matched
    # [code: agent_fusion.py#L179]
    candidates = veh_scores >= 0.05
    # Cost: 3D distance of reference points; pairs failing the box gate
    # get 1e6  [code: agent_fusion.py#L61-L84]
    cost = pairwise_distance(veh_points, inf_points, candidates)
    # Hungarian assignment on the cost matrix
    # [§ Cross-view query matching] [code: agent_fusion.py#L86]
    veh_idx, inf_idx = hungarian(cost)
    # Keep pairs with cost below 1e5 and fuse their content half
    # [code: agent_fusion.py#L106-L110]
    veh_queries = fuse_matched(veh_queries, inf_queries, veh_idx, inf_idx, cost)
    # Append every roadside query that was not accepted
    # [code: agent_fusion.py#L115-L128]
    veh_queries = append_unmatched(veh_queries, inf_queries, inf_idx)
    return veh_queries

# Vehicle side: occupancy fusion  [§ Cross-view occupancy fusion]
# Inputs: veh_occ (T, H, W) and inf_occ (T, H, W) after warping to vehicle frame
def fuse_occupancy(veh_occ, inf_occ, threshold):
    # Binarise both maps at the same threshold (0.1 in the released config)
    # [code: occ_head.py#L541-L542]
    veh_bin = veh_occ > threshold
    inf_bin = inf_occ > threshold
    # Element-wise max of binary maps is a logical OR
    # [code: occ_head.py#L545-L546]
    fused = logical_or(veh_bin, inf_bin)
    return fused

Hyperparameters

Values in the code are from the released e2e config at the commit below. "not stated" means the paper and this config do not give the value.

Results

Author-reported. Setup: DAIR-V2X (about 100 scenes, 28 intersections, 10 Hz, HD map) § Experiments: DAIR-V2X Dataset, image input only, 8 A100 GPUs § Experiments: Implementation. Baselines are re-implemented with BEVFormer, except CoCa3D § Experiments: Agent perception.

Planning tab. 2, averaged over 2.5 s, 3.5 s and 4.5 s:

Detection and tracking, car class tab. 3:

Online mapping, IoU in % tab. 4: No Fusion 6.4 (lane) and 2.7 (crosswalk); Early Fusion 16.7 and 17.8 at 8.19e7 BPS; CoBEVT 15.6 and 16.4 at 2.56e6 BPS; UniV2X 17.8 and 19.8 at 1.47e5 BPS.

Occupancy, IoU in % tab. 5: No Fusion 16.3 (30 x 30 m) and 13.1 (50 x 50 m); UniV2X 22.0 and 26.5.

Ablations (author-reported, DAIR-V2X) app. Fusion modules tab. 6-8: agent fusion alone gives mAP 29.5 and AMOTA 23.9; lane fusion alone gives 17.8 and 19.8 IoU; occupancy fusion alone gives 20.3 and 23.5 IoU. Latency 500 ms tab. 9: UniV2X without flow prediction drops to mAP 25.9; with flow it reaches 28.5 (0 ms: 29.5). Bandwidth tab. 10: mAP rises from 16.5 at 0 Mb/s to 29.5 at 1.0 Mb/s. V2X-Sim (V2V, 20% of data) tab. 11-13: collision rate 2.0% to 1.75%; mAP 7.0 to 7.6; occupancy IoU-f 18.9 to 19.7.

Reliability § Ablation: reliability fig. 4: randomly dropping 10% to 100% of roadside agent queries degrades agent perception gradually. At 100% drop the model is said to be comparable to No Fusion.

Paper vs code

Compared against the official code at commit a6e4f6bd23e3583fce44cfcacad39516ae503e7d (Apache-2.0, https://github.com/AIR-THU/UniV2X, the repository named in the abstract).

• Spatial update: paper uses a three-layer MLP over [Q, R] eq. 3; code uses one linear layer for each of the two query halves agent_fusion.py agent_fusion.py.
• Query fusion: paper says a three-layer MLP produces the fused query that updates the vehicle query § Cross-view query matching; code adds one linear projection of the roadside content half to the vehicle content half and leaves the positional half unchanged agent_fusion.py.
• Matching gate: paper uses Euclidean distance with Hungarian matching; code adds a per-axis gate that rejects pairs whose reference-point offset exceeds the vehicle box size agent_fusion.py agent_fusion.py. The threshold=0.3 argument of forward is not used in its body agent_fusion.py.
• Ego removal: paper says queries inside the ego rectangle are filtered; code removes only the first roadside query inside the rectangle and then stops agent_fusion.py.
• Temporal synchronisation: paper's QueryFlowNet and query flow eq. 2 are not in the code. I searched the detector, fusion, tracker and config directories for "flow" and "latency"; the only hits were occupancy-flow terms and config keys. The released fusion forward (agent_fusion.py#L131-L199) has no time compensation step. The latency ablation tab. 9 cannot be reproduced from this code without other sources.
• Lane fusion: the paper describes matching and fusion as in agent fusion, and also says it concatenates synchronised queries to speed up training. The code concatenates the roadside lane queries with the vehicle lane queries, with a TODO that says so lane_fusion.py. Roadside lane queries are kept if their sigmoid score is above 0.05 lane_fusion.py.
• Occupancy fusion: the paper says max fusion on probabilities, then threshold. The code thresholds each map first, then takes the max occ_head.py. For one shared threshold these two orders give the same result. The released test threshold is 0.1 univ2x_coop_e2e.py.
• Planning post-processing: collision and drivable-area optimisation run only in test mode planning_head.py. Training uses the raw regression output.
• Range and grid: the paper gives a 0.25 m BEV grid over [-50, 50] m and an infrastructure range [0, 100] x [-50, 50] m § Experiments: Implementation. The code uses ranges of 102.4 m and a 200 x 200 BEV, which is 0.512 m per cell univ2x_coop_e2e.py. The occupancy GT grid is 0.5 m over 100 m univ2x_coop_e2e.py.
• Classes: the paper groups detection into four classes (car, bicycle, pedestrian, traffic_cone) § Experiments: DAIR-V2X Dataset. The config lists ten nuScenes-style classes univ2x_coop_e2e.py. The paper reports only the car class.
• Roadside model in the e2e config is frozen univ2x_coop_e2e.py, and the detector has a TODO to unfreeze it multi_agent.py. The paper describes four training stages app. Training. The main e2e config sets one 20-epoch run and loads a stage-1 checkpoint univ2x_coop_e2e.py. The repository also has separate roadside and vehicle configs (for example univ2x_sub_inf_e2e.py and univ2x_sub_vehicle_e2e.py). I did not compare them with the four stages.

Pitfalls

• Section numbers are off. The source sets secnumdepth to 0, so the compiled sections have no numbers. Section citations here use the heading names. Table and figure numbers were counted in source order (tables 1 to 13, figures 1 to 5) and were not checked against a compiled PDF.
• Grid size is inconsistent. A 200-cell BEV over [-50, 50] m is 0.5 m per cell, not 0.25 m. The paper's 0.25 m figure does not match its own range and cell count, so an implementer should pick one and check it against the code's 0.512 m BEV and 0.5 m occupancy grid.
• Baseline off-road reduction. The text says the off-road rate falls by 9.3% against No Fusion § Experiments: Planning results. Table 2 gives 1.08 to 0.89, which is a drop of 0.19 points, or about 17.6% relative. The collision reduction of 61% matches the table (0.89 to 0.34).
• Planning L2 is not best. UniV2X's average L2 (3.48 m) is worse than BEV Feature Fusion (3.30 m) and No Fusion (3.44 m). The paper's text does not claim an L2 win. CooperNaut's lower off-road rate comes with a much larger L2 (5.35 m), which the authors explain as a shorter, more conservative plan.
• Transmission numbers disagree. The tracking table gives UniV2X 6.96e4 BPS, and the text gives 6.94e4 § Experiments: Agent perception. The planning table gives 8.09e5 BPS for UniV2X, which is about 12 times the tracking figure; the paper does not say what the planning figure includes.
• BPS arithmetic in the appendix. The appendix says one 256-dimensional float32 query is 1024 bytes, then says "eight" such floats are 1024 bytes, and that ten queries at 2 Hz give 2.0e3 bytes per second. Ten queries of 1024 bytes at 2 Hz is 2.0e4 bytes per second. Use the per-query size, not the paper's totals, when estimating bandwidth app. Transmission cost.
• No latency compensation in the released code. The paper's latency result depends on query flow eq. 2, which the released code does not contain (see Paper vs code). Plan for no latency compensation unless the flow module is rebuilt from the paper, and note that the paper does not state the flow network's inputs beyond the two query sets.
• Ego filter removes one object. Removing only the first match in the rectangle agent_fusion.py is what the code does; a second ghost detection on the ego is kept. The paper calls this approach "not always optimal" because of position error § Ego identification and removing.
• Stored roadside queries. The appendix says training on roadside queries from training clips caused unstable inference. The fix stores queries from inference mode and trains agent fusion on those app. Training. An implementer who runs fusion on live training clips will see this instability.
• Unused threshold. AgentQueryFusion.forward takes threshold but never uses it. Do not tune it expecting an effect agent_fusion.py.
• Only a few thresholds are in the paper. Matching gate, ego rectangle, lane score cut-off and occupancy threshold are all absent from the text. Use the values in the table above, which come from code.
• Compiled from arXiv v3 source. The LaTeX is version 3 (updated 2024-12-24). The released repository is a later snapshot of the code.

Lineage

• Builds on: BEVFormer for the BEV backbone and UniAD for the query-based planning stack, which the paper reuses in its planning head § Planning output. UniAD is named but has no card.
• Compared against: V2X-ViT (card present), Where2comm, DiscoNet, CoAlign, CooperNaut, CoBEVT, CoCa3D, Late Fusion with AB3DMOT, and a BEV feature fusion baseline that feeds UniAD § Experiments. These are named and not cited to any card.
• Superseded by: not stated.