wikillm

Paper · 2022-12-16 · cs.CV, cs.AI, cs.LG

Learning for Vehicle-to-Vehicle Cooperative Perception under Lossy Communication

Repairs intermediate features that cars share over lossy V2V links. An LC-aware repair network predicts a per-tensor filter that rebuilds damaged features; a V2V attention module fuses ego and neighbour features. Trained and tested on lossy links in simulated OPV2V: 84.1 AP@0.5 on CARLA Towns vs 77.0 for the best intermediate-fusion baseline (author-reported).

Collaborative perceptionAttention mechanismObject detectionv2vlidar3d-object-detectionlossy-communicationpointpillars

arXivPDFCodeMarkdown11 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Self-driving cars can share what their sensors see with nearby cars, so each car gets a fuller picture of the road. The catch is that the wireless link drops or garbles parts of what is sent. Methods built for a clean link break badly when that happens, and in the paper's tests some of them end up doing worse than a car that only uses its own sensors.

The paper adds two parts. The first is a repair step. It takes each received block of shared data and rebuilds the damaged values from their neighbourhood, in the spirit of image denoisers. The second is an attention step. It combines the car's own data with the repaired data from the other cars, so the car can weigh each source. The system is tested in a driving simulator on detecting 3D vehicles from LiDAR. It is trained on damaged data as well, so it learns to cope with the damage.

TL;DR

  • Cooperative perception shares deep features between cars, but the existing intermediate-fusion methods assume an error-free link §I §II.B.
  • Under simulated loss, those methods drop sharply; in the Scheme I training setting, F-Cooper falls from 0.844 to 0.036 AP@0.5 on CARLA Towns tab. I.
  • The proposed LC-aware repair network plus V2V attention module reaches 0.841 AP@0.5 on CARLA Towns under lossy testing, with training on lossy data tab. II.
  • Everything is author-reported on one simulated dataset (OPV2V). The linked code repository holds only a README, so the method cannot be checked against code.

Problem

Intermediate fusion sends deep feature maps from each car to its neighbours §I. If a transmission is lossy, the received maps are incomplete or wrong, and detection quality falls §I fig. 1. The paper says the existing V2V perception work §II.B all assumes ideal communication. It also assumes, for this study, that there are no communication delays and no localisation errors §III.

Lossy links are common in urban traffic. Causes listed include obstacles, Doppler shift, interference and routing failures, and weather §I §II.C.

Method

The method has three parts: per-car feature extraction, a repair network for the received maps, and a fusion module with intra-vehicle and inter-vehicle attention. A detection head then predicts the boxes fig. 2.

Overview §III.A. The ego car builds a graph of the cars within communication range. Every neighbour projects its LiDAR points into the ego frame using the 4×4 transform from its own pose to the ego pose eq. 1. Each car runs a PointPillar backbone, which turns the points into a pillar tensor, scatters it to a 2-D pseudo-image and encodes it §III.A. Neighbours send their feature maps to the ego car, and the maps pass through the repair network and the V2V attention module before the detection head §III.A fig. 2.

LC-aware repair network §III.B. The network has an encoder-decoder shape with skip connections fig. 3. It does not output the repaired map directly. Instead it predicts a kernel for each tensor position and channel eq. 2. Applying that kernel to the k×k neighbourhood of each value gives the repaired map eq. 3. The kernel size is k = 5 because of memory limits §III.B. The design follows denoisers that predict per-pixel filters §III.B.

The repair is trained with an L1 loss against the clean feature map, taken before the damage eq. 4 §III.B.

V2V attention module §III.C. The ego car's own features give the queries for both sub-modules below. The two sub-modules are combined:

  • Intra-vehicle attention. Self-attention over the ego map, using scaled dot-product scores eq. 5.
  • Uncertainty-aware inter-vehicle attention. The neighbour maps provide the keys and values, and the ego map provides the queries eq. 6. The softmax is applied per neighbour and the results are summed over the N neighbours.

The two outputs are summed eq. 7. The text then describes max pooling, average pooling and a 2-D convolution with ReLU as the fusion function F §III.C fig. 4. The text also says the implementation uses two stacked criss-cross attention modules, following CCNet, instead of full dot-product attention, so that each position attends to its row and column only §III.C.

Loss §III.C. Detection uses smooth L1 for box regression and focal loss for classification. The total loss is the detection loss weighted by μ plus the repair loss weighted by λ, where both weights lie in [0, 1] eq. 8.

Training schemes §IV.B. Scheme I trains on ideal data only. Scheme II trains on data with simulated loss. Both use the same settings. Loss is simulated by replacing shared values with uniform noise, chosen from the original feature range, at a rate p drawn uniformly from [0, 1] §IV.B. Section IV.D defines two loss types. "Lossy" selects values from the flattened map. "Ch-Lossy" removes ⌊pC⌋ whole channels §IV.D fig. 6.

Pseudocode. Written from the equations and text, not from code, since the repo has no code. Shapes are (channels, height, width).

Pseudocode
# Lossy V2V cooperative perception: one training step  [§III] [§IV.B]
# Input:  scans, poses (one per CAV), ego (index), gt_boxes
# Output: scalar loss

def training_step(scans, poses, ego, gt_boxes, p):
    # Encode every CAV's LiDAR in the ego frame  [§III.A] [eq. 1]
    feats = [backbone(to_ego(scans[i], poses[i], poses[ego]))
             for i in range(len(scans))]           # (C, H, W) per CAV
    # Clean neighbour features, kept as repair targets  [§III.B] [eq. 4]
    clean = [feats[i] for i in neighbours(ego)]
    # Damage the shared features (training under lossy comms)  [§IV.B]
    received = [lossy_channel(f, p) for f in clean]
    # Predict a per-position kernel and filter each received map  [§III.B]
    kernels = [repair_net(r) for r in received]    # (k*k, H, W), k = 5
    repaired = [local_filter(r, k) for r, k in zip(received, kernels)]
    # Intra-vehicle self-attention over the ego map  [§III.C] [eq. 5]
    intra = scaled_dot_attention(ego_q, ego_k, ego_v)   # (C, H, W)
    # Inter-vehicle attention: softmax per neighbour, then sum  [eq. 6]
    inter = 0
    for r in repaired:
        inter = inter + scaled_dot_attention(ego_q, conv_k(r), conv_v(r))
    # Fuse the summed outputs with pooling and a convolution  [eq. 7]
    fused = fuse(intra + inter)    # F: max pool, avg pool, conv + ReLU
    # Detection loss: smooth L1 on boxes, focal loss on classes  [§III.C]
    det_loss = detection_loss(head(fused), gt_boxes)
    # Repair loss: L1 between clean and repaired neighbour maps  [eq. 4]
    repair_loss = l1_distance(clean, repaired)    # reduction not stated
    # Weighted total; mu = 1.0 and lambda = 0.1 in the experiments  [eq. 8]
    return 1.0 * det_loss + 0.1 * repair_loss

Hyperparameters

NameValueWhereNotes
Detection backbonePointPillar, anchor-based§III.A §IV.Bpillar-based 3D detector
Voxel resolution0.4 m (height and width)§IV.B
OptimizerAdam, initial learning rate 1e-3§IV.Bthe paper cites a decoupled-weight-decay reference for it; weight decay not stated
Learning-rate schedulemultiply by 0.1 every 10 epochs§IV.Btotal epochs not stated
Detection loss weight μ1.0§IV.B eq. 8
Repair loss weight λ0.1§IV.B eq. 8
Repair kernel size k5§III.B"due to memory limitations"
Shared-feature value range[0, 29.5]§IV.Brange used for the replacement noise
Loss probability puniform on [0, 1]§IV.B §IV.Dper sample, per batch or per value not stated
Ch-Lossy channel count⌊pC⌋ channels§IV.D
Evaluation rangex in [-140, 140] m, y in [-40, 40] m§IV.B
CAVs per scene1 to 5§IV.B
AP thresholdsIoU 0.5 and 0.7§IV.B
Criss-cross attention modulestwo, stacked§III.C
Other hyperparametersfollow V2X-ViT's§IV.Bthe values are not restated in this paper
Hardwaretwo RTX 3090 GPUs§IV.B
Frames: train / val / test3,382 / 1,920 / 2,170 (Towns) + 594 (Culver City)§IV.A8,066 frames of the 11,464 in OPV2V; selection rule not stated
Batch size, epochs, weight decaynot stated
Focal-loss α and γnot stated§III.C
Channel widths and depth of repair networknot stated§III.B
Number of 1×1 convolutions and channels in attentionnot stated§III.C
Feature channels C, height H, width Wnot statedthe example in fig. 6 uses C = 9, H = W = 10 for illustration only

Results

Author-reported. Setup: OPV2V, a digital-twin CARLA dataset with two test sets (CARLA Towns and Culver City) §IV.A. The backbone is PointPillar §IV.B. Metrics are AP at IoU 0.5 and 0.7 §IV.B. The tables give AP as decimals (0.926 = 92.6 %). Compared methods are NO Fusion, F-Cooper, V2VNet, OPV2V, CoBEVT and V2X-ViT §IV.B.

Headline (Scheme II, lossy testing, tab. II). Best of the five intermediate-fusion baselines, read from the lossy rows (derived from the table, not stated in the paper):

Test setMetricBest baseline (lossy)This paper (lossy)Source
CARLA TownsAP@0.50.770 (V2X-ViT)0.841tab. II
CARLA TownsAP@0.70.603 (OPV2V)0.705tab. II
Culver CityAP@0.50.795 (CoBEVT)0.846tab. II
Culver CityAP@0.70.586 (CoBEVT)0.663tab. II

Scheme I (trained on ideal data only, tab. I). Under ideal communication, V2VAM (the attention module alone) reaches 0.926 AP@0.5 on CARLA Towns against 0.914 for CoBEVT. Under lossy testing, V2VAM falls to 0.085. The caption mentions a V2VAM+LCRN row, but the table has no such row.

Full tables, as published:

Table I, Scheme I (train on ideal data) tab. I:

MethodCom.Towns AP@0.5Towns AP@0.7Culver AP@0.5Culver AP@0.7
NO FusionIdeal0.6790.6020.5570.471
NO FusionLossy0.6790.6020.5570.471
F-CooperIdeal0.8440.7430.8740.715
F-CooperLossy0.0360.0290.1960.146
V2VNetIdeal0.8740.7120.8550.630
V2VNetLossy0.0240.0140.1020.061
OPV2VIdeal0.8710.7930.8680.745
OPV2VLossy0.0150.0110.0100.006
CoBEVTIdeal0.9140.8360.8770.748
CoBEVTLossy0.0890.0690.2720.202
V2X-ViTIdeal0.8400.7260.8770.720
V2X-ViTLossy0.0830.0540.1070.067
V2VAMIdeal0.9260.8610.8850.785
V2VAMLossy0.0850.0750.0950.070

Table II, Scheme II (train on lossy data) tab. II:

MethodCom.Towns AP@0.5Towns AP@0.7Culver AP@0.5Culver AP@0.7
NO FusionIdeal0.6790.6020.5570.471
NO FusionLossy0.6790.6020.5570.471
F-CooperIdeal0.6770.4940.7580.523
F-CooperLossy0.6770.4920.6560.440
V2VNetIdeal0.7130.4650.7020.408
V2VNetLossy0.7140.4650.7020.409
OPV2VIdeal0.8040.6450.7420.576
OPV2VLossy0.7390.6030.7180.561
CoBEVTIdeal0.8710.7400.8660.688
CoBEVTLossy0.7680.5820.7950.586
V2X-ViTIdeal0.7930.6190.7310.520
V2X-ViTLossy0.7700.5990.7170.511
V2VAM+LCRNIdeal0.8870.7830.8710.709
V2VAM+LCRNLossy0.8410.7050.8460.663

Table III, ablation under Scheme II, lossy testing tab. III:

MethodTowns AP@0.5Towns AP@0.7Culver AP@0.5Culver AP@0.7
NO Fusion0.6790.6020.5570.471
AveFuse (baseline)0.6320.3250.6970.374
V2VAM w/o Intra0.6130.4900.6370.458
V2VAM w/o Inter0.6410.4940.6810.504
V2VAM0.7090.5830.7610.541
AveFuse+LCRN0.6980.4720.7140.558
V2VAM+LCRN0.8410.7050.8460.663

Table IV, two lossy types, Scheme II models tab. IV:

MethodCom.Towns AP@0.5Towns AP@0.7Culver AP@0.5Culver AP@0.7
OPV2VLossy0.7390.6030.7180.561
OPV2VCh-Lossy0.7110.5820.7370.582
CoBEVTLossy0.7680.5820.7950.586
CoBEVTCh-Lossy0.7420.6150.7670.588
V2X-ViTLossy0.7700.5990.7170.511
V2X-ViTCh-Lossy0.7930.6190.7310.520
ProposedLossy0.8410.7050.8460.663
ProposedCh-Lossy0.8520.7230.8510.675

Other results in words: fig. 5 shows detections on Culver City under lossy communication. The text says the five baselines produce false negatives there, and the proposed method finds more of the vehicles §IV.C. The text gives drops for Scheme I on CARLA Towns at AP@0.5 of 80.8 (F-Cooper), 85.0 (V2VNet), 85.6 (OPV2V) and 82.5 (CoBEVT) §IV.C. These are differences in percentage points between the ideal and lossy rows of tab. I (derived, and checked against the table).

Paper vs code

The paper links https://github.com/jinlong17/V2VLC in its abstract, which is why it is treated as the official repository. The clone at commit d75ce6ae9ac6f12a23271db78704eed9d6f64040 holds only a README and an MIT LICENSE. The README says "The code is coming soon" README.md. The repository has no model, loss, training or configuration code.

The method therefore cannot be checked against code. The pseudocode above is written from the paper alone. The abstract points readers to this link for the code, but the repository does not yet contain any. No other repository was searched, since the paper's own link points to the official one.

Pitfalls

  • No code. Batch size, epoch count, weight decay, focal α and γ, channel widths, the depth of the repair network and the number of 1×1 convolutions are all unstated. An implementer must choose them, and the results may not transfer.
  • Softmax placement in eq. 6. The softmax sits inside the sum over neighbours, so each neighbour's weights are normalised separately, not across neighbours. The text does not say this in words.
  • Attention form conflict. Eqs. 5 and 6 are scaled dot-product attention, but the text says criss-cross attention replaces it. Eq. 7 sums the two outputs before fusion, while the text sends each one through max and average pooling and then concatenates them. The paper does not say which version it used. The complexity claim (from O((HW)²) to O(HW(H+W−1))) belongs to the criss-cross version §III.C.
  • "Uncertainty-aware" is not in the equations. No uncertainty is estimated or used in any formula. The only difference is that neighbour maps supply keys and values, while the ego map supplies queries §III.C.
  • Kernel shape. Eq. 2 gives K with shape (k×k)×h×w, with no channel axis. The text calls the kernel per tensor, per channel. The implementation must choose; the text suggests per channel.
  • The ⊛ operator is called a "matrix dot product" in the text §III.B. The description (apply a k×k kernel to each k×k neighbourhood) is local filtering. Boundary handling is not stated.
  • Repair loss reduction. Eq. 4 writes a norm with no order, and the text calls it a tensor-wise L1 distance. Whether it is a sum or a mean over elements is not stated. That choice changes the effective λ = 0.1.
  • Clean targets. The repair loss needs the clean neighbour features before damage §III.B. The paper does not say how that clean map is obtained at training time.
  • Loss probability. p is drawn uniformly from [0, 1], but the paper does not say whether p is drawn per sample, per batch or per value §IV.B.
  • Replacement range. Noise is drawn from the range [0, 29.5], which the paper calls a statistic of its own features §IV.B. The range depends on the backbone and would have to be re-measured for a different one.
  • Table I caption. It mentions a V2VAM+LCRN row that the table does not contain. The text says the full method is best "in Table I". The only method in that table is V2VAM alone.
  • Text and table disagree. The text says V2VAM beats CoBEVT on Culver City by 1.6 points at AP@0.5. Table I gives 0.885 − 0.877 = 0.8 points. The AP@0.7 gap of 3.7 points matches tab. I.
  • "Decrease by X %" is points, not a relative drop. In the text, F-Cooper falls from 0.844 to 0.036, which is 80.8 points. The relative drop is about 95.7 %.
  • Units. Tables use decimals and the text uses percent.
  • Assumptions the headline depends on. The paper assumes no communication delay and no localisation error §III. The results therefore do not cover delay or pose noise.
  • Frame splits. The splits total 8,066 frames of the 11,464 in OPV2V. The selection rule is not stated §IV.A.
  • Single dataset, simulated. All results come from OPV2V in CARLA with one backbone. There is no real-vehicle test.

Lineage

  • Builds on: OPV2V for the intermediate-fusion pipeline and dataset §III.A §IV.A. The PointPillar backbone §III.A and the criss-cross attention, cited to CCNet by the authors, are not linked.
  • Hyperparameters follow V2X-ViT §IV.B.
  • Compared against: CoBEVT, F-Cooper, V2VNet, OPV2V and V2X-ViT, plus the feature-averaging AveFuse baseline that this paper introduces §IV.B.
  • Superseded by: none recorded in this wiki.

Source and license

Adapted from “Learning for Vehicle-to-Vehicle Cooperative Perception under Lossy Communication” by Jinlong Li, Runsheng Xu, Xinyu Liu, Jin Ma, Zicheng Chi, Jiaqi Ma, Hongkai Yu (arXiv:2212.08273), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/jinlong17/V2VLC at commit d75ce6ae9ac6 (MIT). Any quoted code is shown under that license.