PolarFormer: Multi-camera 3D Object Detection with Polar Transformer

In plain words

Most camera-based 3D detectors draw their top-down map on a square grid. PolarFormer argues that each camera actually sees the road as a wedge that spreads out from the car, so it uses a grid of distance and angle instead. Each camera's image columns are turned into rays that run out from the car, and attention decides how far along each ray the objects sit.

The rays from all six cameras are merged into one polar map, at three levels of detail so that both near and far objects get enough resolution. A decoder places objects on that map and predicts their position, size, heading and speed. Heading and speed are measured relative to the direction of the object from the car. The authors report that this beats a strong Cartesian baseline on the nuScenes benchmark.

TL;DR

• PolarFormer detects 3D objects for autonomous driving from six cameras and predicts them on a polar (distance, angle) bird's-eye-view grid instead of the usual Cartesian one §1.
• Image columns feed cross-attention that writes polar rays; three scales of polar maps are refined by deformable attention, and a 6-layer decoder predicts the boxes §3.
• Author-reported on nuScenes val: 39.6 mAP and 45.8 NDS with ResNet-101 tab. 3.
• The temporal variant (PolarFormer-T) is not in the released code, and the numbers depend on extra pretraining for the "improved" setting.

Problem

• Camera-based detectors usually put bird's-eye-view features on a Cartesian grid with perpendicular axes, and convolutional designs are restricted to rectangular grids §1.
• Each camera sees the scene as a wedge whose radial axes are not perpendicular, so the authors argue a polar grid matches the imaging geometry better §1.
• Estimating dense depth to lift images into 3D is costly and error-prone §1. Query-based methods avoid depth but do not model the geometry between objects well §1.
• Polar grids have been used for LiDAR, but those methods rely on convolutions over rectangular grids and local receptive fields §1.
• On a polar map an object's size in cells shrinks with distance, so one scale of polar map does not fit all objects §1 app. scale variance.

Method

The model maps six surround images to a polar BEV representation in four stages §3 fig. 2:

1. Backbone and feature pyramid produce three feature scales per camera §4.
2. A cross-plane encoder, one per scale, lets polar rays attend to the pixels of each image column §3.1.
3. Rays are stacked along azimuth into a polar map per camera, and all cameras are merged onto one shared polar grid §3.1 §3.2.
4. A deformable-attention BEV encoder mixes neighbouring cells and scales §3.3; a decoder with learned queries predicts objects in polar form §3.4.

Camera geometry. For a camera point with depth z_C, the image column x_I gives the azimuth and radius of its ray §3.1:

\phi = \arctan\frac{x^{(I)} - u_0}{f_x}, \qquad \rho = z^{(C)}\sqrt{\left(\frac{x^{(I)} - u_0}{f_x}\right)^2 + 1}

The azimuth depends only on the image column, not on the image row. The radius needs the unknown depth, so the model learns it through attention instead of estimating depth §3.1.

Cross-plane attention. For camera n, scale u and image column w, each polar ray query attends to that column's features with standard multi-head attention, with d_q = d_k = d_v = d_model / h §3.1.

Deformable attention in the BEV encoder and decoder §3.3 §3.4:

\mathrm{MSDeformAttn}(z_q, \hat{x}_q, \{G_u\}) = \sum_{m} W_m \sum_{u}\sum_{k} A_{muqk}\, W'_m\, G_u\!\left(\zeta_u(\hat{x}_q) + \Delta x_{muqk}\right)

The attention weights A are normalised to sum to one over all levels and points; the offsets Δx come from an MLP on the query §3.3.

Orientation and velocity are regressed relative to the azimuth of the object, split into two orthogonal parts §3.4:

\bar\theta = \theta_{ori} - \phi, \qquad \theta_\phi = \sin\bar\theta, \qquad \theta_\rho = \cos\bar\theta

The velocity uses the same split on v_abs and θ_v §3.4. Object sizes are regressed as log l, log w and log h §3.4.

# PolarFormer forward pass for one frame  [§3]
# Input:  images (N, 3, H, W) with N = 6 cameras; intrinsics, extrinsics
# Output: class scores and polar box predictions per decoder query

def polarformer_forward(images, intrinsics, extrinsics):
    # Backbone and feature pyramid give three scales per camera  [§4]
    image_feats = fpn(resnet101(images))
    polar_maps = []
    for scale, feats in enumerate(image_feats):
        # Polar rays attend to the features of each image column  [§3.1]
        rays = cross_plane_encoder(feats, ray_queries[scale])
        # Stack the rays along azimuth into one polar map per camera  [§3.1]
        polar_maps.append(stack_along_azimuth(rays))   # (N, R_s, W_s, C)
    # Merge all cameras onto one shared polar grid per scale  [§3.2]
    bev_polar = [polar_alignment(polar_maps[s], intrinsics, extrinsics)
                 for s in range(3)]
    # Deformable attention mixes neighbouring cells and scales  [§3.3]
    memory = polar_bev_encoder(bev_polar)
    # Learned queries decode the objects in polar coordinates  [§3.4]
    return polar_decoder(memory)

# Polar alignment: sample each camera on the shared polar grid  [§3.2]
# Only visible samples count, and the result is normalised by their number
def polar_alignment(polar_maps, intrinsics, extrinsics):
    # Grid points (rho_i, phi_j) at each height z_k in the ego frame  [§3.2]
    points = polar_grid_points()                    # (R, A, Z, 3)
    weighted_sum = zeros(R, A, C)
    visible_count = zeros(R, A, 1)
    for n in range(num_cameras):
        # Project each point into camera n and read its polar ray index  [§3.2]
        ray_index, visible = project(points, intrinsics[n], extrinsics[n])
        # Bilinear sampling of camera n's polar map  [§3.2]
        samples = bilinear_sample(polar_maps[n], ray_index)
        weighted_sum += (samples * visible).sum(over_height)
        visible_count += visible.sum(over_height)
    return weighted_sum / max(visible_count, 1)     # (R, A, C)

# Polar decoder: learned queries refine (phi, rho, z) reference points  [§3.4]
# Each layer predicts an object: centre, log size, sin/cos heading and velocity
def polar_decoder(memory):
    # Query content and position come from one learned embedding  [§3.4]
    query_content, query_pos = split(query_embedding)
    reference = sigmoid(linear(query_pos))          # (Q, 3) = (phi, rho, z)
    for layer in range(6):
        # Self-attention between queries, then deformable cross-attention [§3.4]
        query_content = self_attention(query_content)
        query_content = deformable_attention(
            query_content, reference[..., :2], memory)
        # Offsets move the reference point in phi, rho and z  [§3.4]
        reference = refine(reference, query_content)
        # Heading and velocity are relative to the azimuth of the centre  [§3.4]
        boxes = box_head(query_content, reference)
    return boxes

Key choices the paper states: the three scales of polar map have radius by azimuth resolutions of 64 by 256, 32 by 128 and 16 by 64 §4. The cross-plane encoder uses three layers per scale and the BEV encoder and decoder use six layers each §4.

Hyperparameters

Values marked "code" are not stated in the paper and come from the official configs or code. Paths are relative to the code repository.

Results

Author-reported. Setup: nuScenes detection, six cameras, mAP and NDS. "Prototype" means a ResNet-101 backbone initialised from an FCOS3D checkpoint. "Improved" means a VoVNet-V2-99 backbone initialised from a DD3D model fine-tuned on extra DDAD15M data §4 tab. 1.

Ablations on the val set tab. 2: Cartesian features with a Cartesian head give 38.1 / 45.5, and the polar feature with a Cartesian head gives 38.5 / 45.0 tab. 2a. Multi-scale polar maps lift the polar model from 39.1 to 39.6 mAP, while the Cartesian model gains almost nothing from multiple scales tab. 2c. The azimuth and radius resolution with the best result is 256 by 64 tab. 2d.

Paper vs code

Official repository, linked from the paper's author line. Searched first: the paper's link. No other repository was used. Code at dadd1bfd213e00ddf3e6c77c4733acc089131142, MIT license, copyright Fudan Zhang Vision Group LICENSE.

• Temporal model missing. The paper describes PolarFormer-T with one history sweep, concatenated in polar BEV space §4. The repository has no temporal module and no temporal config; the README lists PolarFormer-T val results (43.2 / 52.8) with no config and no weights README.md. The temporal-module text in the paper source sits in an \iffalse block, so it does not appear in the compiled paper §3. Not reproducible from this repository.
• Loss and matching. The paper names focal and L1 losses §3.4 but no matching rule and no loss weights. The code uses Hungarian matching with focal cost 2.0 and L1 cost 0.25, classification weight 2.0, box weight 0.25, and GIoU weight 0 polarformer_r101.py polarformer_head.py.
• Localisation loss in Cartesian. The paper says the centre loss is computed in Cartesian coordinates while the prediction is polar app. coordinate choice tab. 4. The code matches: the head converts polar predictions to Cartesian with sin and cos polarformer_head.py. Only orientation and velocity are encoded relative to azimuth util.py. No discrepancy found.
• Polar ray queries. The paper says the ray queries are "introduced" but does not say how they are initialised §3.1. The code uses fixed 2D sine encodings of the column and radius index as the decoder queries, and adds a fixed 2D sine encoding to the image features fpn_trans.py.
• Visibility and normalisation. The paper defines a binary visibility per point and camera and normalises by the sum of visibility over cameras and heights §3.2. The code marks a cell visible only when all of its height samples project inside the image fpn_trans.py. It then averages the height samples per camera fpn_trans.py and divides by the number of visible cameras fpn_trans.py. Visibility also uses a hard-coded half-plane per camera name fpn_trans.py. This differs from the paper's formula.
• Height samples per scale. The paper does not give the number of height samples. The code samples 10, 5 and 3 heights at the three scales, because the height step doubles per level fpn_trans.py.
• Decoder sampling. The paper says the reference points are 3D (phi, rho, z) §3.4. The deformable sampling uses only the first two (phi, rho); z is refined by the regression only polar_transformer.py.
• Augmentation. The paper lists no augmentation §4. The code applies photometric distortion, GridMask (probability 0.7) and BEV rotation, scale and flip polarformer_r101.py polarformer.py. The README reports 39.2 / 46.0 without BEV augmentation, against 39.6 / 45.8 with it README.md.
• Detection range. The appendix proof uses detection range [1, 50] m app. scale variance. The code point cloud range is ±51.2 m in x and y polarformer_r101.py.
• Improved setting. The paper says the improved setting uses DD3D initialisation and extra data §4. The VoVNet config matches this, with use_external=True and a DD3D checkpoint, and a different image normalisation (std about 57) polarformer_vovnet.py polarformer_vovnet.py.

Pitfalls

• Section numbers are hidden in the compiled paper. The source sets secnumdepth to 0, so section and appendix numbers do not print. The citations here count sections in source order (§3.1 is the cross-plane encoder, §3.4 the decoder, §4 the experiments). Check the title if a number does not match.
• Equation numbers are not used. Most equations in the main text are unlabelled in the source, so this card cites sections instead of equation numbers.
• Temporal results are not reproducible here. The PolarFormer-T rows depend on a temporal module that is absent from the code and described only in a disabled source block.
• Improved results depend on extra data. The improved setting needs DD3D pretraining and extra DDAD15M data §4. The licence of that dataset is not stated in this card, so check its terms before using the improved model commercially.
• Matching and loss weights are missing from the paper. An implementer needs the Hungarian matcher and the focal and L1 weights from the code polarformer_r101.py.
• Polar ray queries and positional encodings are underspecified. The paper does not say how ray queries are initialised. The code uses fixed sine encodings, and Table 2b shows fixed sine beats a learnable 2D encoding and a 3D encoding tab. 2.
• Visibility is coarser than the formula. The code's half-plane masks are not the cameras' fields of view, and the cell visibility needs all heights in the image. Expect the visible set to differ from the paper's definition.
• Angle wrap-around. The appendix attributes the slow and weak convergence of a Polar localisation loss to the jump between 0 and 2 pi in azimuth app. coordinate choice tab. 4. The code avoids this by computing the centre loss in Cartesian coordinates.
• Scale-variance argument is a sketch. The appendix proves that a square object's polar area shrinks with distance in a specific range, then says the same holds for arbitrary orientations by treating the object as many small squares app. scale variance. Treat the general claim as heuristic.
• Compute is not reported. The paper text gives no runtime, FPS or memory figures. Each camera runs one cross-plane attention per scale and per image column, and the decoder runs 900 queries over six layers. Measure cost before choosing this model for latency-sensitive use.
• Hard-coded GPU placement. FPN_TRANS calls .cuda() on its tensors, so the model does not run on CPU without edits fpn_trans.py.
• Camera names matter. The visibility masks depend on the nuScenes camera names and their order, and a mismatch silently changes results fpn_trans.py fpn_trans.py.

Lineage

• Builds on: query-based 3D detection with sparse object queries and reference points §1 §3.4; deformable attention for multi-scale features §3.3; FCOS3D checkpoint for initialisation §4; BEVDet4D-style temporal concatenation for the temporal variant §4; DD3D pretraining for the improved setting §4; mmdetection3d as the codebase §4.
• Compared against: BEVFormer and BEVFormer-S, DETR3D, Ego3RT, PGD, FCOS3D, M2BEV and CenterPoint tab. 1 tab. 2 tab. 3.
• Superseded by: not known from content already in this wiki.