BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers

In plain words

A self-driving car has several cameras around it. Older camera-only systems handle each camera's picture on its own and then try to join the detections together afterwards, which works poorly. BEVFormer instead builds one map of the area around the car, seen from above. Each cell of that map is a spot on the road, and it looks up the camera pictures where that spot could appear.

The map is also carried over from one moment to the next. The previous map is moved to match how the car has moved, and it is blended with the current one. That helps with things a single photo shows poorly, such as how fast something is moving and whether an object is partly hidden.

The finished map can feed a 3D object detector or a model that labels the road (drivable area, lanes). The authors report better results than earlier camera-only methods on the nuScenes driving dataset, and results close to some laser-based (LiDAR) systems. Those numbers are the authors' own.

TL;DR

• Input: six surround cameras of nuScenes, plus the previous BEV map §4.1 §3.1.
• A 6-layer transformer encoder turns learnable grid queries (200 × 200) into a BEV map §3.1 §4.2.
• Spatial cross-attention reads camera features where each cell's 3D points project; temporal self-attention reads the previous map §3.3 §3.4.
• Author-reported 56.9 NDS on nuScenes test with a VoVNet-99 backbone that was pre-trained on extra depth data; 51.7 NDS on val with ResNet-101 tab. 1 tab. 2.
• Code is public (Apache-2.0) and matches the paper in most settings; see Paper vs code for the exceptions.

Problem

3D perception for driving needs 3D boxes or a semantic map from several cameras §1. Processing each camera alone and merging afterwards leaves information between cameras unused and hurts accuracy and efficiency §1 §2. A bird's-eye-view (BEV) grid is a common shared representation, but BEV methods that lift image features with predicted depth are sensitive to depth errors, and BEV methods have not clearly beaten other paradigms on 3D detection §1. Stacking BEV features from several past timestamps adds computation and can bring in disturbing information §2 §3.4.

Velocity is also hard to estimate from camera-only detectors without temporal information §4.3.

Method

BEVFormer keeps a fixed set of learnable BEV queries, one per grid cell, and updates them through six encoder layers §3.1 §3.2. The grid is centred on the ego car. Each layer has three parts: temporal self-attention over the current queries and the previous BEV, spatial cross-attention over the multi-camera features, and a feed-forward network §3.1. The previous BEV is passed unchanged to every layer §4.2. The output BEV map feeds a 3D detection head, or a map segmentation head for vehicles, road (drivable area) and lane §3.1 §3.5.

Grid and reference points. A query at grid position (x, y) covers a cell of side s metres. Its real-world coordinates are given by equation 3, with the ego car at the centre §3.2 eq. 3. For spatial attention, the query is lifted to a pillar: N_ref = 4 heights are placed above the cell, and each 3D point is projected into every camera with that camera's matrix T_i §3.3 eq. 4.

x' = (x - \tfrac{W}{2}) \times s; \quad y' = (y - \tfrac{H}{2}) \times s eq. 3

\mathcal{P}(p,i,j) = (x_{ij}, y_{ij}), \; z_{ij}[x_{ij}, y_{ij}, 1]^T = T_i [x', y', z'_j, 1]^T eq. 4

Spatial cross-attention. A query only reads the cameras in which at least one of its projected points lands. It samples image features around each such point with deformable attention, then averages over the views that were hit §3.3 eq. 2. Deformable attention on 2D features is given by equation 1 §2 eq. 1.

\text{DeformAttn}(q, p, x) = \sum_{i=1}^{N_\text{head}} \mathcal{W}_i \sum_{j=1}^{N_\text{key}} \mathcal{A}_{ij} \cdot \mathcal{W}'_i x(p + \Delta p_{ij}) eq. 1

\text{SCA}(Q_p, F_t) = \frac{1}{|\mathcal{V}_\text{hit}|} \sum_{i \in \mathcal{V}_\text{hit}} \sum_{j=1}^{N_\text{ref}} \text{DeformAttn}(Q_p, \mathcal{P}(p,i,j), F_t^i) eq. 2

Temporal self-attention. The previous BEV B'{t-1} is first aligned to the current grid using ego-motion §3.4. Then each query attends to the set {Q, B'{t-1}} with deformable attention, and the sampling offsets are predicted from the concatenation of Q and B'_{t-1} §3.4 eq. 5. For the first frame of a sequence, the history is replaced by a copy of the current queries, which makes this layer ordinary self-attention §3.4.

\text{TSA}(Q_p, \{Q, B'_{t-1}\}) = \sum_{V \in \{Q, B'_{t-1}\}} \text{DeformAttn}(Q_p, p, V) eq. 5

Pseudocode. Written from §3.1 and the equations above. The paper gives no algorithm box, so the layout is ours. Details the code adds are in Paper vs code.

# BEVFormer encoder, one timestamp t  [§3.1]
# Input:  images (6, H, W, 3); prev_bev (H_bev * W_bev, C) or None
# Output: bev (H_bev * W_bev, C)

def bevformer_encode(images, prev_bev, ego_shift, ego_yaw_delta):
    # Per-camera multi-scale features with C = 256  [§4.2]
    features = fpn(backbone(images))
    # Learnable grid queries plus learnable positional embedding  [§3.2]
    queries = bev_queries + bev_positional_embedding
    # Move the history into the current ego frame  [§3.4]
    # [code: projects/mmdet3d_plugin/bevformer/modules/transformer.py#L122-L156]
    if prev_bev is not None:
        prev_bev = shift_and_rotate(prev_bev, ego_shift, ego_yaw_delta)
    for layer in range(6):
        # Temporal self-attention on the current and aligned history  [§3.4]
        queries = temporal_self_attention(queries, prev_bev)
        # Spatial cross-attention into the camera features  [§3.3]
        queries = spatial_cross_attention(queries, features)
        # Feed-forward block  [§3.1]
        queries = feed_forward(queries)
    return queries

# Spatial cross-attention for one BEV cell p  [§3.3] [eq. 2]
# Input:  query (C,) at cell p; features per camera
# Output: aggregated query (C,)

def spatial_cross_attention(query, p, features, cameras):
    # N_ref heights above the cell; the range is given in Pitfalls  [§4.2]
    pillar = [(x_p, y_p, z_j) for z_j in anchor_heights]
    hits = []
    for i, camera in enumerate(cameras):
        # Project the 3D points with the camera matrix  [eq. 4]
        points_2d = project(camera, pillar)
        # Keep the view if a point lies in front of and inside the image
        if any_point_inside_image(points_2d):
            hits.append((i, points_2d))
    total = zeros(C)
    for view, points_2d in hits:
        # Deformable attention around each projected point  [eq. 1]
        total = total + deform_attn(query, points_2d, features[view])
    # Average over the views that were hit  [eq. 2]
    return total / max(len(hits), 1)

Hyperparameters

Results

Author-reported. nuScenes: 6 cameras, 10 classes, metric NDS and mAP §4.1. Baselines are as listed in each table. The paper says the 56.9 NDS test result is comparable to LiDAR-based SSN (56.9) and PointPainting (58.1) §4.3 tab. 1.

nuScenes test tab. 1

* VoVNet-99 was pre-trained on depth estimation with extra data tab. 1 caption.

nuScenes val tab. 2

Waymo val, vehicles tab. 3. The first four numeric columns are APH at 3D IoU 0.5 and 0.7 for LEVEL_1 and LEVEL_2; NDS and AP use the nuScenes metrics.

† NDS with ATE and AAE set to 1 tab. 3 caption.

Joint detection and segmentation, nuScenes val tab. 4. Segmentation is BEV IoU.

* Same heads, with VPN or Lift-Splat as the BEV encoder. † Results taken from their papers tab. 4 caption.

BEV encoder comparison, nuScenes val tab. 5. Memory is GPU memory during training.

† Trained with global attention in spatial cross-attention and fp16 weights; single-scale features, 100 × 100 BEV. ‡ Interacts only with reference points, no learned offsets or weights tab. 5 caption.

Latency, nuScenes val tab. 6. Measured on a V100 with ResNet-101-DCN and 900 × 1600 input.

Ablations, nuScenes val.

• Frames used in training tab. 7: NDS 0.448 (1), 0.490 (2), 0.510 (3), 0.517 (4), 0.517 (5). Four frames is the default because gains level off from 4 on.
• Design switches tab. 8: NDS 0.510 without ego alignment (A), 0.513 without random frame sampling (R), 0.513 without using the history for offsets and weights (B), 0.517 for the full model.
• Robustness to camera extrinsics noise at level 4 fig. 6, app. B: NDS drops 15.2% for BEVFormer-S, 17.3% for BEVFormer-S (point) and 14.3% for BEVFormer. Training with noisy extrinsics drops BEVFormer by 8.9%, and BEVFormer (global) drops 4.0%. At the harshest noise, BEVFormer-S (global) reaches 38.8 NDS against 38.0 for BEVFormer-S app. B.
• Occlusion §4.5 fig. 3: at 0–40% visibility, BEVFormer's recall beats BEVFormer-S and DETR3D by more than 6.0 points.
• Velocity: mAVE on nuScenes test falls from 0.925 (BEVFormer-S, R101) to 0.435 (BEVFormer, R101) tab. 1.

Paper vs code

Code: fundamentalvision/BEVFormer at 66b65f3a1f58caf0507cb2a971b9c0e7f842376c. The first author's repository (zhiqi-li/BEVFormer) only points to it. The paper's abstract links the first repository. Apache-2.0.

• FPN levels. Paper: 1/16, 1/32, 1/64 §4.2. Code: four FPN outputs from the three ResNet stages, which includes 1/8 bevformer_base.py bevformer_base.py.
• Sampling points in spatial attention. Paper: four sampling points per reference point, per head §4.2. Code: 8 points per head and level, split over the four anchors, which gives 2 per anchor bevformer_base.py spatial_cross_attention.py.
• Temporal combination. Paper eq. 5 sums over {Q, B'} eq. 5. Code averages the two streams temporal_self_attention.py.
• Ego alignment. Paper: align the history by ego-motion §3.4. Code also rotates the previous BEV by the change in yaw about the centre (100, 100) transformer.py and shifts the history reference points by the ego translation encoder.py. A comment in the code calls that shift a bug kept to reproduce the paper's numbers.
• History sampling in training. Paper: three extra samples from the past 2 s §3.6. Code: takes the four previous dataset indices, drops one at random, and keeps the current frame nuscenes_dataset.py. The 2 s window matches only if the index spacing is about 2 s; not checked here.
• Loss and matching. Paper: L1 for boxes only; no classification loss or matching method named §3.5 app. A.3. Code: focal classification loss and Hungarian matching bevformer_base.py.
• Training additions not in the paper. Linear warm-up, minimum LR ratio, gradient clipping at 35, GridMask, photometric distortion, frozen stage 1 and frozen BN bevformer_base.py bevformer_base.py bevformer_base.py bevformer_base.py.
• Small model. The paper's BEVFormer-S is the static variant §4.2. The repo's bevformer_small.py is a different, lighter model: 150 × 150 BEV, three layers, C5 only, queue length 3 bevformer_small.py bevformer_small.py. No config for the paper's BEVFormer-S is listed.
• Missing configs. No Waymo BEVFormer config and no V2-99 config in the repo; the paper's Waymo settings and VoVNet-99 numbers cannot be reproduced from it §4.2 tab. 1.

No discrepancies found in the optimiser, learning rate, epochs, batch size, query count or BEV grid size.

Pitfalls

• Test-set number depends on extra pre-training. The 56.9 NDS row uses a VoVNet-99 initialised from a depth-estimation model trained on extra data tab. 1 caption. The ResNet-101 row (53.5 NDS test, 51.7 val) is the like-for-like comparison tab. 1 tab. 2.
• Anchor heights are not the stated range. Paper: uniform from −5 m to 3 m §4.2. Code: four cell centres in that range, −4.5, −2.17, 0.17 and 2.5 m encoder.py encoder.py. Use the code's values to reproduce results.
• Cell index versus centre. Eq. 3 uses x and y with no half-cell offset eq. 3. The code uses cell centres, (i + 0.5) s, so grid positions are offset by half a cell (0.256 m at s = 0.512 m) encoder.py.
• Projection needs fp32. The code forces fp32 and disables TF32 during point sampling encoder.py. Points with depth ≤ 1e-5 are masked out by clamping the divisor encoder.py.
• Rotation units. The code passes can_bus[-1] to a rotation that works in degrees, and the dataset writes yaw in degrees can_bus.ipynb transformer.py. Check the units if you rebuild the ego data.
• Inference is stateful. Frames must run in time order. The history resets when the scene token changes or when video_test_mode is off bevformer.py. The first frame of each scene has zeroed ego-motion nuscenes_dataset.py.
• Training history has no gradient. Past frames are encoded under torch.no_grad, so gradients flow only through the current frame nuscenes_dataset.py bevformer.py §4.2.
• Spatial attention is a Python loop. Per-camera gathering and scatter run as nested Python loops over batch and cameras, which limits throughput spatial_cross_attention.py spatial_cross_attention.py. Latency is 130 ms per frame on V100 for the default tab. 6.
• Memory cost. Global attention in spatial cross-attention needs about 36 GB against about 20 GB for deformable attention, and the paper says dense attention is unaffordable at this scale tab. 5 app. A.4.
• Temporal cost is small but not zero. The history pass adds a no-grad forward per training sample, and inference needs the previous BEV kept in memory §3.6 bevformer.py.
• Waymo and V2-99 are not reproducible from the repo. Only nuScenes configs exist .
• Figure and table numbers. This card cites tables and figures in TeX order (1 to 8 and figs. 1 to 9). Check against the compiled PDF before publishing.

Lineage

• Builds on: Deformable DETR, arXiv:2010.04159 (the deformable attention is adapted from it) §2 temporal_self_attention.py.
• Compared against (named, no cards): DETR3D, FCOS3D, PGD, DD3D, VPN, Lift-Splat, CaDNN, PointPillars, SSN, PointPainting, CenterPoint tab. 1 tab. 3 tab. 4.
• Map head is built on Panoptic SegFormer's mask decoder (named, no card) §3.5.