V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer

In plain words

A self-driving car sees the road with its own sensors, and a large vehicle or a corner can hide things from it. Cars nearby, and sensors mounted high on poles at the roadside, can see some of what it cannot. The hard part is combining these views when the positions are a little wrong and the messages arrive late.

This paper proposes a neural network built on attention, a way for each piece of information to decide how much to take from every other source. It uses two kinds of attention. One treats cars and roadside sensors differently, because they are mounted at different heights and are different kinds of devices. The other looks at each scene at several zoom levels, so the model can cope with small position errors. The authors also built a simulated driving dataset with these imperfect conditions, since no public one existed. They report that the model finds other vehicles more reliably than earlier fusion methods when positions and timing are imperfect.

TL;DR

V2X-ViT is a Transformer that fuses LiDAR feature maps from connected vehicles and roadside infrastructure for 3D vehicle detection §3. Its two attention modules, heterogeneous multi-agent self-attention and multi-scale window attention, are stacked in three encoder layers §3.2 §4.2. The authors compare it with no fusion, late and early fusion, and four intermediate-fusion methods on their simulated V2XSet dataset, under perfect and noisy settings §4.1 tab. 1. All numbers are author-reported. The MIT-licensed code fixes several details the paper leaves open (see Paper vs code).

Problem

A single vehicle's LiDAR suffers from occlusion and sparse returns at long range §1. Sharing data between vehicles gives more viewpoints, but most prior vehicle-to-vehicle methods ignore roadside sensors, which sit higher and are often less occluded §1. Mixing vehicles and infrastructure is hard because their sensors differ in type, noise level, mounting height and modality §1. GPS localization errors and asynchronous sensing corrupt the coordinate transforms and delay the shared information §1. The paper assumes imperfect localization and feature-transmission delay throughout §3. Early fusion needs a lot of bandwidth, late fusion keeps little scene context, and intermediate fusion (sharing learned features) is the compromise the authors build on §2.

Method

The pipeline has five stages: metadata sharing, feature extraction, compression and sharing, the V2X-ViT fusion, and a detection head §3. The ego vehicle receives features from the agents within its communication range, aligns them to its current pose, and fuses them with the Transformer §3.1 fig. 2.

1. Metadata sharing. Each connected agent shares its pose, extrinsics and type (vehicle or infrastructure). One connected vehicle is the ego, and the agents form a directed graph. Each agent projects its LiDAR points into the ego pose before feature extraction. The paper assumes this metadata arrives synchronized §3.1.
2. Feature extraction. An anchor-based PointPillar encoder turns each agent's point cloud into an H × W × C bird's-eye-view feature map §3.1.
3. Compression and sharing. A stack of 1 × 1 convolutions reduces the channel count to C′ ≪ C before transmission, and the receiver projects it back to C §3.1. The default compression rate is 32 §4.2. See Paper vs code for what the released code does.
4. Spatial-temporal correction (STCM). The received features are warped to the ego's current pose with a differentiable 2-D affine transform and bilinear sampling. An ROI mask marks pixels that were padded by the warp §3.1 app. B.1.
5. Delay-aware positional encoding (DPE). A time-delay encoding is added to each agent's features before the Transformer §3.2.3 eq. 8 eq. 9 eq. 10.
6. V2X-ViT encoder. Three encoder layers. Each layer applies heterogeneous multi-agent self-attention (HMSA), then multi-scale window attention (MSwin), then an MLP, with layer normalization before each part §4.2 app. B.3 app. tab. T2.
7. Detection head. Two 1 × 1 convolutions predict, for each predefined anchor, a classification score (object or background) and a 7-value box regression (x, y, z, w, l, h, yaw). Regression uses smooth L1 loss and classification uses focal loss §3.1.

HMSA §3.2.1 eq. 1–eq. 7. Nodes have two types (vehicle, infrastructure) and edges have four (V-V, V-I, I-V, I-I) §3.2.1. Attention is computed only between features at the same spatial position across agents, which keeps spatial cues intact §3.2.1. Queries, keys and values use a linear projection chosen by node type and head eq. 3 eq. 4. The attention score for each agent pair also uses a weight matrix chosen by edge type eq. 2. The message from each neighbour is projected by its node type and edge type eq. 6 eq. 7. The output is a node-type-specific linear map of the weighted sum of messages eq. 1.

MSwin §3.2.2 app. B.2. The module runs k parallel branches on one agent's map. Branch j partitions the map into non-overlapping windows of size jP, so the experiments use windows 4, 8 and 16 §4.2. Smaller windows use more heads: 16, 8 and 4 app. tab. T2. Each branch adds a learned relative-position bias app. B.2. The branch outputs are fused with a split-attention module [app. B.2]. The paper states that, unlike Swin, MSwin needs no masking, padding or cyclic shift §3.2.2. Its cost is linear in the number of pixels app. B.2 app. tab. T1.

DPE §3.2.3 eq. 8–eq. 10. A sinusoidal encoding of the delay is computed over the feature channels, passed through a linear layer, and added to every spatial position of the agent's features.

# V2X-ViT forward pass for one ego vehicle  [§3.1] [app. B.3]
# Input:  points[i] per agent (M agents), delays[i], poses[i], types[i]
# Output: class scores and box regressions on the ego grid

def v2x_vit_forward(points, delays, poses, types):
    # 1. Encode each agent's LiDAR as a BEV feature map  [§3.1]
    features = [pillar_encoder(cloud) for cloud in points]
    # 2. Compress before transmission, then decompress  [§3.1] [§4.2]
    features = [compress_decompress(f) for f in features]
    # 3. Warp each map from its capture pose to the current ego pose,
    #    and build a mask for padded pixels  [§3.1] [app. B.1]
    features, masks = warp_to_ego(features, poses, delays)
    # 4. Add the delay-aware encoding to every agent  [§3.2.3] [eq. 10]
    features = [add_delay_encoding(f, d) for f, d in zip(features, delays)]
    # 5. Repeat the fusion block L times  [app. B.3]
    for layer in range(num_layers):
        features = hmsa(features, types, masks)
        features = mswin(features)
        features = mlp(features)
    # 6. Predict boxes and scores on the fused map  [§3.1]
    return detection_head(features[ego_index])

# Heterogeneous multi-agent self-attention (HMSA), one spatial cell  [eq. 1]
# Input:  feats (M, C) one vector per agent; types (M,) vehicle or infra
# Output: out (M, C)

def hmsa(feats, types, valid):
    # Node-type-specific query, key and value per agent  [eq. 3] [eq. 4]
    queries = node_type_dense_q(feats, types)   # (M, C)
    keys = node_type_dense_k(feats, types)      # (M, C)
    values = node_type_dense_v(feats, types)    # (M, C)
    # Score each pair (receiver i, sender j) with an edge-type matrix  [eq. 2]
    # Scale by sqrt(C) in the paper; the code scales by sqrt(dim_head)
    scores = pair_scores(queries, keys, types)  # (M, M)
    # Padded senders get no weight  [app. B.1]
    scores = where(valid, scores, -inf)
    # Normalize over senders j for each receiver i  [eq. 1]
    weights = softmax(scores, over=senders)     # (M, M)
    # Messages depend on the sender and on the edge type  [eq. 6] [eq. 7]
    messages = pair_messages(values, types)     # (M, M, C)
    # Weighted sum, then a node-type-specific output projection  [eq. 1]
    return node_type_dense_out(sum_j(weights * messages), types)

# Multi-scale window attention (MSwin) for one agent's map  [app. B.2]
# Input:  feat (H, W, C); window sizes [P, 2P, 4P]; heads [16, 8, 4]
# Output: fused (H, W, C)

def mswin(feat, window_sizes, heads_per_branch):
    branch_outputs = []
    for window, heads in zip(window_sizes, heads_per_branch):
        # Cut the map into non-overlapping windows, with no padding  [§3.2.2]
        windows = partition(feat, window)       # (count, window*window, C)
        # Self-attention inside each window, with a relative-position bias
        # taken from a (2*window - 1)^2 table  [app. B.2]
        attended = windowed_attention(windows, heads, window)
        branch_outputs.append(merge(attended, window))
    # Adaptive fusion of the branches by split attention  [app. B.2]
    return split_attention(branch_outputs)  # (H, W, C)

# Delay-aware positional encoding (DPE)  [eq. 8] [eq. 9] [eq. 10]
# Input:  feat (H, W, C) for one agent; delay (integer, see Pitfalls)
# Output: feat with the delay encoding added at every position

def add_delay_encoding(feat, delay):
    # Sinusoid of the delay across the C channels, initial value  [eq. 8]
    base = sinusoid(delay, channels=C)     # (C,)
    # Linear projection of the sinusoid, added to the feature  [eq. 9] [eq. 10]
    encoding = linear(base)                # (C,)
    return feat + encoding

Hyperparameters

The table lists values the paper states, values only the released code sets, and values the paper leaves unstated. Code paths are relative to the repository root at code_commit.

Results

Author-reported. Setup: V2XSet, a simulated dataset made with CARLA and OpenCDA, with 6,694 / 1,920 / 2,833 train / validation / test frames §4.1. Metric is average precision (AP) for vehicles at IoU 0.5 and 0.7, on LiDAR only. Every intermediate method uses the same PointPillar backbone and also receives infrastructure data §4.1 §4.2 tab. 1.

Table 1 tab. 1: 3D detection on V2XSet, perfect and noisy settings.

Headline claims in the text §4.3: under the noisy setting, V2X-ViT is 23 and 21.2 AP points above no fusion at AP@0.5 and AP@0.7. Under the perfect setting it is 3.8 and 1.7 points above the best intermediate baseline. The inference time is 57 ms for the full model and 28 ms for a one-layer variant, on a Tesla V100 tab. 4.

Component ablation, noisy setting tab. 2:

Time delay effect of DPE, AP@0.7, noisy setting tab. 3:

Inference time on Tesla V100, with AP@0.7 perfect / noisy tab. 4:

Speed bands, AP@0.7 noisy setting app. tab. T3: V2X-ViT scores 0.693 for objects under 20 km/h, 0.634 for 20 to 40 km/h and 0.488 above 40 km/h. DiscoNet scores 0.639, 0.580 and 0.420 in the same bands.

Road types, AP@0.7 noisy setting app. tab. T4:

Other results are described in words only. Under pose noise up to 0.2 m and 0.4 degrees, V2X-ViT loses under 3 percent AP@0.7, while the other intermediate methods lose at least 6 percent §4.3 fig. 4. Under transmission-size experiments, early fusion falls to 28 percent, and V2X-ViT stays strong up to 128× compression §4.3 fig. 5 app. D.

Paper vs code

Code paths are relative to the repository root at code_commit. The repository is MIT-licensed LICENSE.

1. Compression. Paper: 1 × 1 convolutions, default rate 32 §3.1 §4.2. Code: a 3 × 3 convolution encoder with batch norm and ReLU, and a decoder that ends with a further 3 × 3 convolution [code: v2xvit/models/sub_modules/naive_compress.py#L5-L24]. The released config sets compression: 0, which disables the module [code: v2xvit/hypes_yaml/point_pillar_v2xvit.yaml#L84] [code: v2xvit/models/point_pillar_transformer.py#L32-L34]. The README says to set it to 32 only after perfect-setting training README.md.
2. Order of STCM and DPE. Paper's block equation adds DPE after STCM [app. B.3] (unnumbered equation after "Architectural Configurations"). Code adds the delay encoding to the features before the spatial warp [code: v2xvit/models/sub_modules/v2xvit_basic.py#L164-L168].
3. Attention scale. Paper: 1/√C eq. 2. Code: dim_head ** -0.5, with dim_head 32 hmsa.py [code: v2xvit/hypes_yaml/point_pillar_v2xvit.yaml#L124].
4. Softmax over heads. Paper's equation applies one softmax to the concatenation of the head scores eq. 1. Code applies a separate softmax per head, over the sender axis hmsa.py.
5. Relation matrices. Paper: W in R^(C×C) for each edge type §3.2.1. Code: one dim_head × dim_head matrix per relation and head [code: v2xvit/models/sub_modules/hmsa.py#L30-L33].
6. Split-attention width. Code hard-codes 256 channels [code: v2xvit/models/sub_modules/mswin.py#L104]. Paper does not state the width.
7. Evaluation range. Paper: x in [-140, 140] m, y in [-40, 40] m §4.2. Code: [-140.8, -38.4, -3, 140.8, 38.4, 1] [code: v2xvit/hypes_yaml/point_pillar_v2xvit.yaml#L45].
8. Localization noise. Code adds Gaussian noise to x, y and z, and adds one heading sample to yaw only. Roll and pitch are untouched. The random seed is reset on every call basedataset.py. The paper describes positional and heading noise without naming the angles it perturbs.
9. Time delay and transmission. Paper: 100 ms delay in the noisy setting §4.2. Code's sim mode gives a delay of the overhead divided by 100 ms, so the default reproduces 100 ms basedataset.py. The real mode adds a uniform overhead, a fixed transmission time from 1.06 Mb at 27 Mbps, and a backbone delay [code: v2xvit/hypes_yaml/point_pillar_v2xvit.yaml#L13-L15]. The transmission time does not change with the compression rate, while the paper's appendix ties transmission time to feature size app. D.
10. Learning-rate schedule. Paper: decay ×0.1 every 10 epochs §4.2. Code: MultiStepLR with milestones at epochs 15 and 50, 60 epochs total [code: v2xvit/tools/train_utils.py#L199-L202] [code: v2xvit/hypes_yaml/point_pillar_v2xvit.yaml#L158-L161].
11. Losses. Paper names smooth L1 for regression and focal loss for classification §3.1. Code uses focal loss with alpha 0.25 and gamma 2 [code: v2xvit/loss/point_pillar_loss.py#L72-L73], a smooth L1 with beta 1/9 [code: v2xvit/loss/point_pillar_loss.py#L16], and a sine-difference term for yaw point_pillar_loss.py. The yaw term is not in the paper.
12. Extra inputs. Code appends three channels to each agent's features: velocity, time delay and an infrastructure flag [code: v2xvit/models/point_pillar_transformer.py#L78-L107]. The paper's equations use only features, delays and node types.
13. Delay embedding. Paper: a learnable embedding, sinusoid-initialized [eq. 8] §3.2.3. Code initializes the table with a sinusoid, then sets emb.requires_grad = False on the module [code: v2xvit/models/sub_modules/v2xvit_basic.py#L40-L59]. That assignment targets the module, not the weight tensor, so the table may still train. This is a reading of the code; it was not run.
14. Sinusoid exponent. Paper's equation uses exponent 2c/C with c = 2k [eq. 8]. Code uses 2k/C for embedding dimension 2k [code: v2xvit/models/sub_modules/v2xvit_basic.py#L42-L44].
15. Unstated in paper, set in code. Batch size 2, 60 epochs, dropout 0.3, Adam weight decay 1e-4, anchor size, and points per voxel [code: v2xvit/hypes_yaml/point_pillar_v2xvit.yaml#L19-L43].

Pitfalls

• Equation typos. The block equation in the appendix writes MSwin(HMSA(LN(z_0))) for every layer, which reads z_0 where z_(ℓ−1) is meant app. B.3. The MSwin formula places V inside the softmax, softmax((QK^T/√d + B)V), which is not standard attention. Code applies the usual softmax(scores) × V [code: v2xvit/models/sub_modules/mswin.py#L60-L71]. Follow the code here.
• Percentages are absolute AP points. "21.2% gain" is 0.614 − 0.402 §1 §4.3. A relative gain would be about 53 percent.
• Ablation text vs table. The text says HMSA raises AP@0.7 by 6.6 points §4.5. In Table 2 the step that adds HMSA goes from 0.548 to 0.601, a difference of 5.3 points. The 6.6 figure matches 0.548 to 0.614, which also includes DPE. Treat the text's number as unverified.
• Delay units. The paper leaves Δt's unit unstated. Code quantizes delay to 100 ms basedataset.py, then multiplies the index by RTE_ratio = 2. The code's comment says the embedding then counts 50 ms steps v2xvit_basic.py [code: v2xvit/hypes_yaml/point_pillar_v2xvit.yaml#L116]. Use the code's units when you implement this.
• Compression ambiguity. The paper's compression is 1 × 1 and the code's is 3 × 3 with BN and ReLU. Which one a reproduction uses changes the transmitted size and the parameter count. The released default turns it off.
• Masked softmax. Masked pairs are set to −∞ before the softmax [code: v2xvit/models/sub_modules/hmsa.py#L135-L137]. A receiver with no valid sender would produce NaN. The paper does not discuss this case, and this compile did not check whether the ROI mask can produce it.
• Cost of HMSA. The complexity analysis covers MSwin only app. B.2 [app. tab. T1]. HMSA attends across all agent pairs at each cell, and its cost in the number of agents is not stated. The appendix says the gain becomes marginal beyond four collaborators, with compute still growing linearly app. A.
• Simulation only. The dataset is simulated (CARLA, OpenCDA) and the authors note bias and generalization issues for real-world use §5. Latency numbers are for the V100 setup in tab. 4.
• Dataset license. The paper does not state a license for V2XSet. The README says the data is hosted on a separate download site README.md. The code is MIT, but the dataset license is not known from this compile.
• Figure, table and equation numbers. Equation numbers 2 to 8 were counted from the LaTeX source, where \numberthis tags the attention equations, and table numbers follow the order of captions in the source. Check them against the rendered PDF before citing them elsewhere.
• Training recipe. The README trains in four stages (perfect with compression off, perfect with compression on, noisy with real delays, and finally tests on both settings) README.md. The paper's two-stage description §4.2 omits the compression step.

Lineage

• Builds on: intermediate fusion for V2V perception, specifically OPV2V (attentive fusion of shared features). The shifted-window attention of Swin is the design MSwin is compared against §2 app. B.2. The heterogeneous graph transformer is cited as inspiration for HMSA §2.
• Compared against: No Fusion, Late Fusion, Early Fusion, F-Cooper, OPV2V, V2VNet and DiscoNet tab. 1.
• Superseded by: not stated in the paper, and not known from the wiki.