Fine-Grained Representation for Lane Topology Reasoning

In plain words

A self-driving car has to know which lane leads into which, especially at intersections. Earlier methods describe each lane with one summary vector and decide connections by comparing those vectors. When two lanes only meet at their ends, the summaries of the wrong pair can look alike, and the model picks the wrong connection.

TopoFG describes each lane with a row of small pieces, one for each point along the lane. Where those pieces sit is guided by two hints: a rough top-down map of where lane pixels are, and the fixed order of the points along the lane. Only the first and the last piece of each lane are then compared with the other lanes to decide connections. During training, noisy copies of the true lanes are added so that the connection labels stay fixed, instead of shifting with the matching step. The idea is that more precise lane ends make connection decisions easier to get right.

TL;DR

• Each lane is a sequence of fine-grained queries, one per point, instead of a single instance query §3.3 fig. 1 fig. 2.
• The queries start from a BEV lane mask (spatial prior) and learnable point-order embeddings (sequential prior) §3.2 eq. 1-4.
• Connections come from the first and last query of each lane through an MLP over lane pairs, trained with Hungarian matching plus a fixed-label denoising group §3.4 eq. 6-7.
• Author-reported on OpenLane-V2 with a ResNet-50 backbone: OLS 48.0 on subset_A and 45.4 on subset_B, against TopoLogic's 44.1 and 42.3 tab. 1.
• An official repository is linked, but it has no LICENSE file and only a subset-A checkpoint is listed in its README (see Paper vs code).

Problem

Lane topology reasoning predicts which lanes connect, together with lane centerlines and traffic elements §1. Existing methods treat each lane as one unit with a single query for all of its points, and they decide connections from the overall similarity of two lane queries §1 §2. The authors argue that two lanes that truly connect may touch only at their boundary points, so their overall similarity is low and the connection is missed. The lane-level query can also be too weak for lanes with complex shapes §1 fig. 1.

Method

Overview §3.1 fig. 2. Multi-view camera images go through a CNN backbone. Deformable attention, following BEVFormer, turns the multi-scale image features into BEV features. These are extended to several scales and fed to three modules: the Hierarchical Prior Extractor (HPE), the Region-Focused Decoder (RFD) and the Robust Boundary-Point Topology Reasoning module (RBTR). A lane head then decodes the queries into centerline points §3.5.

Notation §3.1. The camera-feature tensor is written as F_I with dimensions named C, H, W and D, where C is the number of camera views, H and W are the image height and width, and D is the channel count §3.1. The BEV grid size is not stated in the paper.

Step 1, HPE mask §3.2. A set of learnable queries Q^L interacts with the BEV features through a Mask2Former-style module, giving refined queries Q^R. A mask head turns Q^R into one BEV lane mask M per query §3.2. Each lane query is therefore tied to a predicted region of the BEV grid.

Step 2, HPE mask weights §3.2 eq. 1. Each BEV cell gets a weight A. Cells with probability at most tau keep their probability. Cells above tau get the constant alpha, which the paper calls a scaling factor for high-confidence regions. The paper sets tau = 0.3 and alpha = 1.0 §4.2.

\bm{A} = \bm{M} \cdot \mathbb{I}[\bm{M} \le \tau] + \alpha \cdot \mathbb{I}[\bm{M} > \tau] eq. 1

Step 3, HPE spatial prior §3.2 eq. 2. A sine-cosine position encoding P of the BEV grid is summed with the weights A, then normalised. The result is added to Q^R. The paper names the normalisation N_norm without defining it.

\bm{Q}^{\text{pos}} = \mathcal{N}_{\text{norm}}({\bm{P}^\top \cdot \bm{A}}) + \bm{Q}^R eq. 2

Step 4, HPE sequential prior §3.2 eq. 3. A second set of learnable queries Q' stands for the points along a lane. Each point gets an ordered index 1 to k (k = 11 §4.2). The index is positionally encoded and passed through an MLP F, then added to Q'.

\bm{Q}^{\text{seq}} = \mathcal{F}(PE(I)) + \bm{Q}' eq. 3

Step 5, RFD fine-grained queries §3.3 eq. 4. For lane i and point t, the spatial prior of the lane and the sequential prior of the point are summed to give one query.

\bm{Q}^F_{i,t} = \bm{Q}^{pos}_i + \mathcal{F}(\bm{Q}^{\text{seq}}_t) eq. 4

Step 6, RFD decoder layer §3.3 eq. 5. The decoder has m = 6 layers §4.2. In each layer the queries first attend across lane instances, which captures interactions between lanes at the same point index. Then they attend within each lane, which refines the point structure §3.3. A deformable cross-attention step then reads the BEV features. Its reference points R come from the predicted lane mask, instead of being random or fully learnt as in TopoLogic §3.3.

Q^{F, m+1}_{i,t} = \mathcal{A}_{\text{def}}\left(Q^{F, m}_{i,t}, \, \bm{F}_B, \, \bm{R}_{i,t} \right) eq. 5

Step 7, RBTR boundary points §3.4. Only the first and last query of each lane are kept as the boundary-point features f_start and f_end §3.4. The paper's stated reason is that lane topology depends on endpoint relations.

Step 8, RBTR pairwise connectivity §3.4 eq. 6-7. For every ordered pair (i, j), the end feature of lane i and the start feature of lane j are concatenated and passed through a shared MLP. The output is the connectivity logit S_ij. A sigmoid of S gives the similarity topology.

r_{i \rightarrow j} = \text{Concat}(f_i^{end}, f_j^{start}) eq. 6

\bm{S}_{i \rightarrow j} = \mathcal{F}(r_{i \rightarrow j}) eq. 7

Step 9, geometric topology §3.4. The Euclidean distance between the boundary points of all lane pairs gives a distance matrix. A mapping from TopoLogic turns each distance into a score. The similarity and geometric matrices are summed into the final adjacency used for topology reasoning §3.4.

Step 10, Hungarian matching and denoising §3.4. Predicted lanes and ground truth are matched one to one with Hungarian matching §3.4. The paper argues that the supervision matrix then changes between epochs. To fix this, each ground-truth instance is copied G = 5 times into noisy denoising queries §3.4 §4.2. The adjacency for these queries is the ground-truth matrix expanded into a block-diagonal form, which stays fixed. Denoising queries are used only in training. At inference only the vanilla queries are used §3.4.

Step 11, lane head §3.5. Each query decodes to a single key point of the lane, following the TopoLogic design §3.5.

# TopoFG forward pass with the topology head  [§3.1-§3.4] [eq. 1-7]
# Input:  camera images; per lane k = 11 ordered points (training only)
# Output: lane points (B, N, k, 3); topology logits (B, N, N)

def topofg_forward(images):
    # Build a BEV grid from the cameras and extend it to several scales  [§3.1]
    bev_maps = multi_scale_bev(images)           # 4 scales
    # Predict one BEV lane mask per learnable lane query  [§3.2]
    lane_queries, masks = mask_decoder(bev_maps)  # (B, N, d), (B, N, H, W)
    # Keep the probability where it is at most tau, else use alpha  [eq. 1]
    weights = where(masks <= tau, masks, alpha)   # (B, N, H, W)
    # Spatial prior: mask-weighted sine-cosine grid encoding  [eq. 2]
    spatial = normalize(grid_pe.T @ weights) + lane_queries  # (B, N, d)
    # Sequential prior: point-index encoding plus learnable point queries  [eq. 3]
    sequential = mlp(index_pe) + point_queries               # (k, d)
    # Fine-grained query of lane i at point t  [eq. 4]
    queries = spatial[:, :, None] + sequential[None, None]   # (B, N, k, d)
    # Reference points sampled inside the predicted lane mask  [§3.3]
    refs = sample_mask_points(masks > tau, k)                # (B, N, k, 2)
    for layer in range(6):
        # Attend across lanes at each point index  [§3.3]
        queries = self_attn_across_lanes(queries)
        # Attend along the points of each lane  [§3.3]
        queries = self_attn_along_lane(queries)
        # Deformable cross-attention to the BEV features  [eq. 5]
        queries = deform_attn(queries, bev_maps, refs)
    # Keep only the first and last query of each lane  [§3.4]
    start = queries[:, :, 0]                                 # (B, N, d)
    end = queries[:, :, -1]                                  # (B, N, d)
    # Connectivity logit for every ordered pair (i, j)  [eq. 6-7]
    pair = concat(end[:, :, None], start[:, None])           # (B, N, N, 2d)
    topology_logits = mlp(pair)                              # (B, N, N)
    return queries, topology_logits

# Inference-time adjacency: similarity plus geometric term  [§3.4] [code]
# Input:  topology logits (B, N, N); predicted lanes (B, N, k, 3)
# Output: adjacency (B, N, N)

def adjacency_at_inference(topology_logits, lanes):
    # Gap between the end of lane i and the start of lane j  [§3.4] [code]
    end_points = lanes[:, :, -1]                              # (B, N, 3)
    start_points = lanes[:, :, 0]                             # (B, N, 3)
    gap = abs(end_points[:, :, None] - start_points[:, None]).sum(-1)
    # Map the gap to a score in (0, 1]; zero the self-pairs  [§3.4] [code]
    geometric = exp(-gap ** 2 / w) * (1 - eye(N))             # w = 11.5275
    # Sum the learnt similarity and the geometric term  [§3.4]
    return sigmoid(topology_logits) + geometric

Hyperparameters

Values below come from the paper unless marked "code only". Config values are for subset_A (the subset_B config differs where noted in Paper vs code).

Results

Author-reported. Setup: OpenLane-V2 v2.1.0 devkit §4.1 §4.2; subset_A with 7 views and subset_B with 6 views §4.1; ResNet-50 backbone; 24 epochs on 8 NVIDIA A100 GPUs §4.2. Baselines as listed. The paper does not report seeds or variance.

Table 1 tab. 1: OpenLane-V2 subset_A and subset_B. The "+" values are the gains over the second-best method. TopoMLP's row was evaluated with its released model.

Table 2 tab. 2: OpenLane-V2 lane-segment task. Gains are over the second-best method.

Table 3 tab. 3: contribution of the three modules (subset_A).

Table 4 tab. 4: HPE sub-modules (subset_A). LP = local sequential prior; GP = global spatial prior.

Table 5 tab. 5: RFD sub-modules (subset_A). FQI = fine-grained query initialisation; SRP = sampled reference points.

Table 6 tab. 6: RBTR sub-modules (subset_A). BTR = boundary-point topology reasoning; DTR = denoised topology reasoning.

Qualitative §4.5 fig. 3. Predictions are drawn in the bird's-eye view, with lanes in orange and connections as blue arrows. The paper says TopoNet, TopoMLP and TopoLogic miss lanes and make wrong connections, especially at intersections, while TopoFG recovers more complete lanes. This is a visual judgement on selected scenes, not a metric.

Paper vs code

Repository: https://github.com/GXmmm18/TopoFG at commit 81e07ef01d706a928932e9d866d6f57d8eff6360. It is the repository the paper links (\links in the source) and its README names the same three authors. I read the subset_A and subset_B configs, topofg_head.py, sgnn_decoder.py, mask2former_head.py, relationship_head.py, topofg.py, util.py and the transform pipeline. I did not read the BEVFormer modules or the evaluation code. The README lists a checkpoint and config for subset_A only.

Discrepancies found:

• Distance metric. The paper says Euclidean distance between boundary points §3.4. The code uses the L1 sum of absolute coordinate differences topofg_head.py.
• Geometric constants. The code uses P = 2 and w = 11.5275, both hard-coded topofg_head.py. The paper gives no values for the mapping, only "following TopoLogic" §3.4.
• Where the geometric term is used. The code adds the geometric term only at inference, with both weights fixed to 1 topofg_head.py. Training supervises the similarity logits alone topofg_head.py. The paper does not say which of these it does.
• Boundary features. The paper's eq. 6 concatenates only the two boundary features §3.4. The code also appends the predicted, normalised 3D coordinate of each boundary point sgnn_decoder.py.
• Sequential prior. The paper's eq. 3 and eq. 4 use an MLP F. The code uses one linear layer for the index encoding topofg_head.py topofg_head.py, and adds the sequence query to the lane query without a further F topofg_head.py.
• Spatial-prior normalisation. The paper's N_norm is not defined. The code divides by the maximum absolute value over the whole batch tensor topofg_head.py. Each sample's prior therefore depends on the other samples in the batch.
• Alpha. The code does not read alpha. It writes 1 into every cell above tau topofg_head.py. With the paper's alpha = 1.0 the result is the same.
• Reference points. The paper says reference points are sampled from RoI regions §3.3. The code takes 11 mask pixels by evenly spaced index in a sorted list, not at random util.py. If a mask is empty it falls back to a fixed grid util.py. The z component is set to 0 util.py.
• Gradient stopping. The mask query is detached before it enters the lane queries topofg_head.py. The mask prediction is detached before it feeds the query generator topofg.py. The paper does not mention either.
• Lane classification. The code classifies each lane by averaging the class logits of its 11 point queries topofg_head.py. The paper does not say how lanes are scored.
• Traffic-element topology. The code has a lane-to-traffic-element topology head, a traffic-element DETR head, and a loss for both topofg_head.py sgnn_decoder.py. The paper's method describes only lane-to-lane topology, though it reports TOP_lt §3.4 tab. 1.
• Subset_B differences. The subset_B config uses 2D points (pts_dim = 2), a different z-range, and alpha = 0.75 for both topology focal losses, against 0.25 for subset_A. It also drops the front-view crop topofg_r50_8x1_24e_olv2_subset_B.py topofg_r50_8x1_24e_olv2_subset_B.py topofg_r50_8x1_24e_olv2_subset_B.py. The paper mentions none of these.
• Input size. The paper's sizes §4.2 cannot be checked from the repository. The subset_A pipeline crops and scales, but the raw image size is not in the repo.

Consistent with the paper: 200 lanes, k = 11, 6 decoder layers, G = 5 denoising groups, tau = 0.3, AdamW with learning rate 2e-4 and weight decay 0.01, 500 warm-up steps, 24 epochs, batch size 1 per GPU, and 7 and 6 views topofg_r50_8x1_24e_olv2_subset_A.py topofg_r50_8x1_24e_olv2_subset_A.py topofg_r50_8x1_24e_olv2_subset_A.py.

No LICENSE file exists in the repository, so code_license is recorded as none. I quote no code. Everything from the code is paraphrased with citations.

Pitfalls

• Section numbers are not printed. The source sets secnumdepth to 0, so the compiled paper shows no numbers. The § numbers on this card count sections in source order: Introduction §1, Related Work §2, Method §3 (§3.1 to §3.5), Experiments §4 (§4.1 to §4.5). Check a cited section against the arXiv PDF before relying on the number.
• The notation for the camera-feature tensor is inconsistent. §3.1 uses C for the camera count and D for the channel count, which is the reverse of common convention. The code uses 256 channels topofg_r50_8x1_24e_olv2_subset_A.py.
• Ablation text overstates the tables.
  • §4.4 says adding the global spatial prior improves all metrics §4.4. In Table 4 the full model has lower TOP_ll (26.1 against 26.7) and TOP_lt (26.6 against 27.3) than the LP-only row tab. 4.
  • §4.4 says the RFD with sampled reference points gives the best result on every metric §4.4. In Table 5 the FQI-only row has higher TOP_ll (28.6 against 26.8) tab. 5.
• Denoising is only partly specified. The paper gives G = 5 and says the noisy queries come from ground truth §3.4 §4.2. The noise scales come from the code, not the paper mask2former_head.py mask2former_head.py. The config's dn_label_noise_ratio (0.2) is passed in but never read mask2former_head.py. lane_noise_scale is set to 0.5 and never used mask2former_head.py.
• The mask gradient path is stopped. Lane and topology losses do not reach the mask head through the spatial prior, the reference points or the instance query (see Paper vs code). An implementer who follows eq. 2 and eq. 5 literally, without these stops, will get different gradients from the code.
• No variance is reported. The paper gives no seeds, run counts or spread for any ablation row tab. 3 tab. 4 tab. 5 tab. 6. Differences of a few tenths of a point, such as the 0.3 OLS step from LP-only to both priors in Table 4 (45.1 to 45.4), cannot be judged from the card.
• Unstated costs. The paper gives no memory, parameter count or latency. The pairwise head scores all N × N = 200 × 200 lane pairs, and the mask-based sampling loops over masks and batch items in Python util.py.
• Hard-coded values. P = 2, w = 11.5275 and lambda = 1 are fixed in the inference path topofg_head.py. Commented-out lines show learnable versions topofg_head.py.
• Subset_B is a different setup. The B config changes point dimension, z-range and focal alpha, so the B numbers are not a simple replication of the A setup topofg_r50_8x1_24e_olv2_subset_B.py topofg_r50_8x1_24e_olv2_subset_B.py. Only the A checkpoint is listed in the README.
• README inconsistency. The README's weights table labels the TopoFG checkpoint as TopoLogic. Its results table for subset_A matches the paper's numbers.
• The qualitative result is anecdotal. Figure 3 shows selected scenes with no per-scene metric, so it cannot be checked against the tables §4.5 fig. 3.

Lineage

• Builds on: TopoLogic, for the geometric mapping §3.4, the lane head §3.5 and the baseline in every ablation tab. 3. The paper also replaces TopoLogic's learnt reference points with mask-guided ones §3.3.
• Builds on: BEVFormer for BEV features and Mask2Former for the mask head §3.1 §3.2. Neither has a card on this wiki yet.
• Compared against: TopoMLP (official model), TopoNet (arXiv:2304.05277), LaneSegNet (arXiv:2312.16108), MapTR, STSU and VectorMapNet tab. 1 tab. 2. Benchmark: OpenLane-V2 (arXiv:2304.10440).
• Superseded by: not known from content already in this wiki.