Cross-view Transformers for real-time Map-view Semantic Segmentation

In plain words

A self-driving car has several cameras around it, and it needs a single top-down map of where the road and the vehicles are. Usually the software first estimates how far away each pixel is, then projects the pixels onto the ground. Those depth estimates are error-prone, especially far away.

This paper skips the depth step. The network learns a grid of map cells and lets each cell look at the camera images with an attention mechanism. To tell the attention where each camera is and which way it points, every camera is given a position code computed from its calibration. The network learns on its own how the map cells line up with the image pixels.

The model is small and simple, and the authors report that it runs in real time and matches the best earlier methods on vehicle segmentation on nuScenes.

TL;DR

• Encoder-decoder for map-view (top-down) segmentation from six cameras. An EfficientNet-B4 encodes each image; cross-view attention lifts the features into a 25×25 map embedding, which a small decoder upsamples to 200×200 §3.2 §4.
• Each camera's keys carry a positional embedding from its intrinsics and extrinsics, so no explicit depth or homography is computed §3.1.
• Author-reported on nuScenes vehicle segmentation (100 m × 100 m at 50 cm, "Setting 2"): IoU 36.0, with 5M parameters and 35 FPS on an RTX 2080 Ti tab. 1.
• Paper and code differ in several places (per-head dimension, input resolution, query formation, normalisation, feature order). See Paper vs code.

Problem

Online map-view perception needs the layout of the road and vehicles around the car. Inputs are camera images in camera frames, and outputs are on a ground-plane grid §2. Earlier models use explicit geometry: they predict depth or assume a planar homography, then pool features into the map §2. The authors argue that explicit depth is error-prone at range and that depth-based projection is a rigid bottleneck §1. Models that learn the mapping from data without geometry (for example VPN) must learn camera calibration implicitly in their weights §2. The paper targets a model that uses calibration as an explicit input yet still avoids explicit geometry, with fast inference §1.

Method

The network takes N = 6 images with their intrinsics K and extrinsics (R, t) relative to the ego vehicle §3. It predicts a binary mask y ∈ {0,1}^(h×w×C) in the orthographic map frame §3. Here the output is a single vehicle channel on a 200 × 200 grid covering 100 m × 100 m around the ego vehicle §4 §5.

Geometric starting point

The projection from world to image is written as a homogeneous relation up to scale eq. 1. Without depth, a world point is ambiguous. The authors keep the geometry as a cosine similarity between a pixel's unprojected ray direction and the vector from the camera centre to a map point eq. 2. They then replace the geometric terms with learned embeddings §3.1.

Cross-view attention

§3.1 eq. 3

• Camera-aware key embedding. Each pixel's ray direction, unprojected with the inverse intrinsics and the inverse extrinsics and taken from the camera centre, is passed through an MLP shared across cameras. The result is a D-dimensional embedding δ with D = 128 §3.1.
• Keys and values. Keys combine δ with the image feature φ. Values are the image features §3.1.
• Map-view queries. Each map cell has a learned embedding c. The query for camera k subtracts a camera-location embedding τ_k, which an MLP computes from the camera centre §3.1.
• Attention. Each map cell attends over the pixels of all cameras in one softmax. The logit is a cosine similarity between query and key, and the pair compared is always from the same camera eq. 3 §3.1.
• Refinement. The map embedding is refined by repeated cross-attention and MLP blocks. Each refinement queries more precise image locations fig. 2 §1.

# Cross-view attention at one feature scale  [§3.1] [eq. 3]
# Input:  map_state (B, D, H, W), with H = W = 25
#         features (B, N, C, h, w), N = 6 cameras
#         inv_intrinsics (B, N, 3, 3), inv_poses (B, N, 4, 4)
# Output: map_state (B, D, H, W)

def cross_view_attention(map_state, features, inv_intrinsics, inv_poses):
    # Pixel rays from each camera centre, in the ego frame  [§3.1] [eq. 1]
    pixels = homogeneous_pixel_grid(h, w)                    # (3, h, w)
    rays = inv_poses @ homogeneous(inv_intrinsics @ pixels)  # (B, N, 4, hw)
    # Camera centre t_k: translation column of the inverse pose  [§3.1]
    centres = inv_poses[..., -1:]                            # (B, N, 4, 1)
    # Camera-aware key embedding, MLP shared over cameras  [§3.1]
    key_pos = mlp_direction(rays - centres)                  # (B, N, D, hw)
    # Keys: positional embedding plus image features  [§3.1]
    keys = normalise(key_pos + features)                     # (B, N, D, hw)
    # Values: image features  [§3.1]
    values = features                                        # (B, N, C, hw)
    # Queries: map embedding minus the camera-location embedding  [§3.1]
    camera_pos = mlp_camera(centres)                         # (B, N, D)
    queries = normalise(map_state[:, None] - camera_pos[..., None, None])
    # Cosine logit, query of cell q against keys of camera n  [eq. 3]
    logits = cosine_pairs(queries, keys)                     # (B, Q, N, hw)
    # One softmax per map cell, over all cameras and pixels  [§3.1]
    weights = softmax(flatten(logits, over=(N, hw)))         # (B, Q, N*hw)
    # Weighted sum of the image features across all cameras  [§3.1]
    attended = weights @ flatten(values, over=(N, hw))       # (B, Q, C)
    # Project to D; the projection is not stated in the paper  [§3.1]
    attended = project(attended)                             # (B, Q, D)
    # Residual update with an MLP block  [§1] [§3.2]
    updated = map_state + attended
    return updated + mlp(updated)

Architecture

§3.2 fig. 2

• Image encoder. One shared EfficientNet-B4, fine-tuned from pretrained weights, applied to each camera §4. It yields features at two scales: 28×60 (8× downscaling) and 14×30 (16×) §4.
• Map embedding. A learned tensor of size 25×25×128, shared across all inputs at the start §4.
• Scales. One cross-view attention block per scale. The paper says it starts from the lowest resolution and then repeats at the higher one §3.2 §4. See Paper vs code, item 3.
• Decoder. Three blocks, each a bilinear 2× upsample followed by a convolution, take the 25×25 map to 200×200 §4. A final head produces the logits §3.2.

# Full network  [§3.2] [§4] [fig. 2]
# Input:  images (B, N, 3, H_img, W_img), inv_intrinsics, inv_poses
# Output: vehicle logits (B, 1, 200, 200)

def forward(images, inv_intrinsics, inv_poses):
    # Shared image encoder, one pass per camera  [§4]
    # 8x features are 28 x 60, 16x features are 14 x 30  [§4]
    # (B, N, C, 28, 60) and (B, N, C, 14, 30)
    feature_8x, feature_16x = backbone(images)
    # Learned map-view embedding, 25 x 25 x D  [§4]
    map_state = learned_map_embedding            # (B, D, 25, 25)
    # Lowest resolution first, then the finer scale  [§3.2] [§4]
    for feature in [feature_16x, feature_8x]:
        map_state = cross_view_attention(
            map_state, feature, inv_intrinsics, inv_poses)
    # Three bilinear 2x upsample + conv blocks  [§4]
    return decoder(map_state)                    # (B, 1, 200, 200)

Hyperparameters

Values stated in the paper are cited to the paper. Values marked code come from the official repository only, at the commit in the front matter.

Code-only values (not in the paper, from the official repository at the commit above):

Results

All numbers are author-reported. Setting 1 is 100 m × 50 m at 25 cm; Setting 2 is 100 m × 100 m at 50 cm. Both use IoU §5. Table 1 compares single-timestep models only, and the authors compare to FIERY (static) tab. 1.

Vehicle segmentation on nuScenes tab. 1

The authors state that the two methods are on par, with a slight edge to this model in Setting 2 and to FIERY in Setting 1 §5.1. The stated speed is 4.5× faster inference tab. 1. Training takes 32 GPU hours against 96 for FIERY §5.1.

Vehicle and driveable-area segmentation tab. 2

The caption says the top rows are nuScenes Setting 2 and the bottom rows are Argoverse tab. 2. The Argoverse dataset is not in the official code.

Ablations of cross-view attention tab. 3

Setting 2, vehicle IoU, one component changed per model, trained from scratch §5.2.

Camera-aware positional embedding tab. 4

Setting 2, vehicle IoU §5.3.

Distance and camera dropout fig. 3 fig. 4

• Distance. Both this model and FIERY have near-identical error modes. Accuracy falls with distance. This model degrades more slowly at short range and slightly under-performs FIERY at long range §5.4 fig. 3.
• Camera dropout. Dropping 0, 1, 2 or 3 random cameras gives vehicle IoU of 36.0, 31.2, 27.6 and 22.5 (values from the plot data in the LaTeX source). The authors describe the drop as roughly linear in the visible area §5.5 fig. 4.

Paper vs code

Official code: https://github.com/bradyz/cross_view_transformers, linked from the abstract. Read at commit 4de6e641397ef1ffde996d7549f7f988e49156f7. The repository is MIT-licensed, copyright Brady Zhou LICENSE. The paper's authors are Brady Zhou and Philipp Krähenbühl; the repository owner matches the first author.

Differences between the paper and the code. Each item lists both sources.

1. Input size. Paper: 224 × 448 §5. Code resizes each image to 480 wide and 270 tall, then drops the top 46 rows, giving 224 × 480 [code: cross_view_transformer/data/transforms.py#L126-L132] [code: config/data/nuscenes.yaml#L28-L31]. The paper's feature sizes 28×60 and 14×30 §4 fit 480 wide, not 448, so the 224 × 448 figure is probably a text error.
2. Head dimension. Paper: 4 heads, d_head = 64 §4. Code: 4 heads, 32 per head cvt.yaml, scaled by 32^-0.5 [code: cross_view_transformer/model/encoder.py#L118].
3. Scale order. Paper: start from the lowest resolution (14×30), then the higher one §3.2 §4. Code: features are returned in the order listed in the config, reduction_2 (28×60) first, then reduction_4 [code: config/model/cvt.yaml#L19] efficientnet.py efficientnet.py [code: cross_view_transformer/model/encoder.py#L331-L335]. The code therefore starts at the higher resolution.
4. Extra encoder layers. Paper: two cross-attention blocks, and nothing after them §4. Code: after each cross-attention, two ResNet bottleneck blocks at the map width D cvt.yaml [code: cross_view_transformer/model/encoder.py#L10] [code: cross_view_transformer/model/encoder.py#L312] [code: cross_view_transformer/model/encoder.py#L331-L335].
5. Attention block internals. Paper: attention and MLP blocks §1, with no norms or residual placement stated. Code: LayerNorm on the query, key and value inputs, a LayerNorm before and after the MLP, an MLP with GELU and 2× hidden width, and a skip from the map state [code: cross_view_transformer/model/encoder.py#L123-L130] [code: cross_view_transformer/model/encoder.py#L165-L173].
6. Map-view query. Paper: the query is the map embedding minus the camera embedding, and the paper says the map side "can no longer rely on exact geometric inputs" §3.1. Code: the query also adds a fixed embedding of the ego-frame coordinates of each map cell, at each cross-attention block, and the learned state encoder.py [code: cross_view_transformer/model/encoder.py#L274].
7. Embedding networks. Paper: MLPs for the direction embedding and the camera embedding §3.1. Code: a single 1×1 convolution with no bias for each, applied to the homogeneous 4-vector [code: cross_view_transformer/model/encoder.py#L216-L218] [code: cross_view_transformer/model/encoder.py#L244-L256]. Because these projections have no bias, the code's direction embedding is a linear map of the difference between the ray point and the camera centre.
8. Normalisation. Paper: cosine similarity, normalising the key sum δ + φ and the query c − τ eq. 3. Code: normalises the positional part of the key before adding the features, and normalises the query separately encoder.py [code: cross_view_transformer/model/encoder.py#L266-L268].
9. Keys and values. Paper: keys are δ + φ and values are φ §3.1. Code: both keys and values pass the image features through BatchNorm, ReLU and a 1×1 convolution first encoder.py encoder.py.
10. Logits. Paper: cosine similarity eq. 3. Code: dot product of LayerNorm-ed linear projections, scaled by 32^-0.5, with no cosine normalisation encoder.py. The code's logits are not the paper's cosine logits.
11. Loss. Paper: focal loss, with no γ or α stated §3.2 §4. Code: sigmoid focal loss with γ = 2 and α = −1 center_loss.yaml losses.py, averaged over pixels whose visibility is at least 2 [code: cross_view_transformer/losses.py#L52-L53], plus a centre-heatmap loss with weight 0.1 center_loss.yaml [code: cross_view_transformer/losses.py#L59-L80]. The centre loss is not in the paper. The code says α = −1; fvcore's convention for a negative α is to disable α weighting, but this was not checked against the fvcore source.
12. Training length. Paper: 30 epochs §4. Code: 30001 steps [code: config/config.yaml#L42]. The epoch count cannot be derived without the dataset size used in training, which is not stated.
13. Learning rate. Paper: not stated §4. Code: 4e-3 peak, one-cycle config.yaml config.yaml.
14. Decoder. Paper: three bilinear 2× upsample + conv layers §4. Code adds a residual from the encoder output at each block [code: cross_view_transformer/model/decoder.py#L27-L36] [code: config/model/cvt.yaml#L49], and a head of conv3×3, BN, ReLU and conv1×1 [code: cross_view_transformer/model/cvt.py#L29-L33].
15. Pretraining. Paper: pretrained EfficientNet-B4, fine-tuned §4, with the pretraining data not stated. Code: from_pretrained('efficientnet-b4') [code: cross_view_transformer/model/backbones/efficientnet.py#L56].
16. Metric. Paper: IoU, with no threshold stated §5. Code: pixel-level IoU at thresholds 0.4 and 0.5 on sigmoid outputs, counts accumulated over the whole dataset metrics.py [code: config/metrics/visibility_metrics.yaml#L1-L4].
17. Settings and datasets. Paper: Setting 1 and Setting 2, plus Argoverse §5. Code: a 200 × 200 grid over 100 m, which matches Setting 2 [code: config/data/nuscenes.yaml#L19-L24]. No Setting 1 configuration and no Argoverse loader were found in the repository.
18. Data split. Paper: not stated. Code: nuScenes split files are in the repository train.txt val.txt.
19. Speed benchmark. Paper: 35 FPS on an RTX 2080 Ti §1. Code: a benchmark script runs 1000 forward passes with batch size 1 and optional mixed precision benchmark.py benchmark.py. The GPU and whether the paper used this protocol are not stated.

Pitfalls

• Attention scale is unspecified. The paper's logit is a cosine similarity in [-1, 1] eq. 3, and no temperature is stated. A softmax over raw cosines is very flat. The code uses a learned projection with a 32^-0.5 scale instead (item 10). Decide the scale explicitly and check it against the code.
• Which cameras attend to which. The paper states that each query compares only with keys from the same camera, through the camera-specific query §3.1 eq. 3. The softmax runs jointly over all cameras and pixels. This is easy to get wrong if the softmax is taken per camera.
• Features at 224 × 480 vs 224 × 448. Use the feature sizes (28×60, 14×30) to set the input size. The code's crop and resize are described in item 1. Intrinsics must be rescaled with the same resize and crop [code: cross_view_transformer/data/transforms.py#L135-L139].
• Which map positions the query sees. Item 6 is the largest deviation from the paper's text. The code injects ego-frame coordinates into every query. Following the paper's description alone gives a different network.
• Missing pieces that the code needs. Norms, residual placement, the MLP width, the output head and the map embedding initialisation are not in the paper. The code is the only source for them (items 5, 14, Hyperparameters table).
• Scale order. The paper and code process the two feature scales in opposite orders (item 3). Check which one a reimplementation follows, and reproduce the ablation if it matters.
• Metric. Use the pixel-level, dataset-accumulated IoU of the code, and record the threshold. The paper does not give one, and per-frame averaging would give different numbers.
• Visibility masking. The loss and the metric ignore pixels with visibility below 2 in the code losses.py visibility_metrics.yaml. The paper does not mention this, so IoU numbers depend on it.
• Speed numbers. The 35 FPS and 4.5× figures come from the authors' measurements on an RTX 2080 Ti §1 tab. 1. The batch size, precision and timing protocol are not stated in the paper. The code's benchmark uses batch size 1 benchmark.py.
• Argoverse results cannot be reproduced from the code. No Argoverse loader or configuration is in the repository tab. 2.
• Figures. Figures are described in words only. fig. 1 shows the teaser and fig. 2 the architecture. fig. 3 and fig. 4 are the distance and dropout plots. fig. 5 shows qualitative results and fig. 6 shows attention maps for map points §5.6 §5.7.

Lineage

• Builds on: explicit geometric map-view methods, including Lift-Splat-Shoot (Philion and Fidler), OFT (Roddick et al.) and PON (Roddick et al., 2020), which the paper compares against §2 tab. 1 tab. 2. The paper replaces their explicit depth or pooling with learned cross-attention.
• Related methods compared: VPN §2 tab. 1, STA tab. 1, FIERY (static) §5.1 tab. 1, and Monolayout tab. 2. None has a card here.
• Image backbone: EfficientNet-B4, cited in the paper as tan2019efficientnet §4. No card here.
• Superseded by: not known from this wiki's content.