HDMapNet: An Online HD Map Construction and Evaluation Framework

In plain words

Self-driving cars need a map of the road right around them: where the lane lines are, where the curbs and crosswalks lie. Building such maps ahead of time, by driving survey cars and labelling the results by hand, is expensive and goes stale. HDMapNet asks whether the car can work out this local map on its own from what it senses right now.

The model takes pictures from several cameras around the car, and/or a LiDAR point cloud. It turns everything into a single top-down view of the road, then labels each small cell as lane divider, lane boundary, crosswalk or background. Cells that belong to the same line are grouped using learned "embeddings", and each line is turned back into an ordered list of points by also predicting which way the road runs at each cell.

The authors found that cameras and LiDAR are good at different things, and that combining them works best. They also propose ways to score such maps, both per pixel and per line, so that results from different methods can be compared.

TL;DR

• Predicts vectorized local HD map elements (lane divider, lane boundary, pedestrian crossing) in bird's-eye view from six surrounding cameras, LiDAR, or both §1 §3.
• Camera features are mapped to BEV with a learned per-camera MLP and then warped with camera extrinsics; LiDAR uses a PointPillars-style encoder §3.1 §4.1.
• Three heads: semantic segmentation, instance embedding, and lane direction; lines are recovered by clustering and direction-guided connection §3.1.
• Author-reported on nuScenes: camera-LiDAR fusion gets 44.5 IoU and 30.6 mAP over all classes, versus 32.9 IoU and 22.7 mAP for cameras alone tab. I tab. II.
• Several details that matter for re-implementation are absent from the paper but present in the official code, and some paper and code values disagree (see Paper vs code).

Problem

Traditional HD semantic maps are built offline from SLAM point clouds and annotated by people. The authors argue this needs heavy human effort and resources to annotate and maintain, which limits scalability §1 §2. They propose learning a local semantic map online from on-board sensors, for real-time motion prediction and planning, and say it is not meant to replace global HD map reconstruction §1.

Assumptions: the map is local (around the ego vehicle) and built from a single frame of sensor input §3. Camera-only view transformation is hard because depth is missing, which the authors name as the core difficulty §1.

Method

HDMapNet is a single network with up to four learned parts: a perspective-view image encoder φ_I, a neural view transformer φ_V, a pillar-based point-cloud encoder φ_P, and a map-element decoder φ_M. The model variants are HDMapNet(Surr) (cameras only), HDMapNet(LiDAR) and HDMapNet(Fusion) §3.1 fig. 2.

Image branch §3.1 fig. 3:

1. Each of the N_m surrounding cameras is embedded by one shared network φ_I into a perspective-view feature map §3.1.
2. A view transformer maps every perspective-view feature map to the camera's top-down coordinate system. For each camera, an MLP models the relation between each top-down cell and every perspective-view pixel §3.1 eq. 1. The paper describes this as a neural transformation; the MLP is shared across channels §4.1.
3. The top-down camera features are transformed to the ego (bird's-eye) frame with the camera extrinsics. The paper states the final image feature is the average of the N_m camera features §3.1.

LiDAR branch §3.1: points are grouped into pillars, and a PointNet with a 64-dimensional layer aggregates the points of each pillar §3.1 §4.1 eq. 2. The pillar features are then encoded by a convolutional network into a BEV feature map §3.1.

Decoder §3.1: a fully convolutional network with three branches, all fed with the image BEV feature, the LiDAR BEV feature, or both concatenated §3.1.

• Semantic branch: per-pixel class prediction trained with cross-entropy §4.1.
• Instance-embedding branch: a discriminative loss pulls embeddings of one instance towards their mean and pushes instance means apart §3.1 eq. 3 eq. 4 eq. 5.
• Direction branch: lane directions are discretized into N_d classes spread uniformly on the unit circle. Each lane pixel is labelled with two directions (both the forward and backward direction of the lane) and both are treated as positive labels. Pixels off the lanes get a zero label and no gradient §3.1. The value of N_d is not stated in the paper §3.1.

Vectorization at inference §3.1: cluster the instance embeddings with DBSCAN, apply non-maximum suppression to remove redundant pixels, then greedily connect the remaining pixels into ordered vectors using the predicted directions. The compiled paper gives no pseudocode for this step, so the steps are described from the code (see the pseudocode below).

# Inference-time vectorization  [§3.1]
# Input:  class probs (K, H, W), embedding (D, H, W), direction logits
# Output: list of ordered polylines per map class

def vectorize(probs, embedding, direction):
    # Each cell takes its most likely class
    # [code: postprocess/vectorize.py#L51]
    labels = argmax(probs, axis=0)
    # Top two direction bins per cell, -1 is no direction
    # [code: postprocess/vectorize.py#L28-L36]
    directions = top_two_directions(direction)
    polylines = []
    for cls in map_classes:
        mask = labels == cls
        # Drop components of 100 px or fewer
        # [code: postprocess/cluster.py#L281-L285]
        mask = remove_small_components(mask, min_pixels=100)
        # Standardise embeddings, then DBSCAN
        # [code: postprocess/cluster.py#L159]
        ids = dbscan(standardise(embedding[:, mask]), eps=1.5, min_samples=50)
        for instance_id in unique(ids):
            if instance_id == -1:
                continue
            # Keep cells that win non-maximum suppression across the line
            # [code: postprocess/vectorize.py#L45-L73]
            ridge_points = directional_nms(cells_of(instance_id), probs)
            # Grow an ordered line from a random start, both ways
            # [code: postprocess/connect.py#L87-L94]
            line = connect_by_direction(ridge_points, directions, step=7)
            polylines.append(line)
    return polylines

Instance-embedding loss as written in eq. 3 to eq. 5. Margins and weights are the paper's values §4.1; see Paper vs code for the official implementation's differences.

# Discriminative instance loss  [eq. 3] [eq. 4] [eq. 5]
# Input:  embedding (D, H, W), instance ids (H, W), 0 is background
# Output: scalar loss, alpha = beta = 1, delta_v = 0.5, delta_d = 3.0

def discriminative_loss(embedding, instance_ids):
    # One mean embedding per labelled instance  [eq. 3]
    centroids = []
    var_terms = []
    for inst in unique(instance_ids[instance_ids != 0]):
        cells = embedding[:, instance_ids == inst]
        centroid = mean(cells, axis=1)
        centroids.append(centroid)
        # Pull each cell towards its centroid, beyond delta_v  [eq. 3]
        dist = norm(cells - centroid[:, None], axis=0)
        var_terms.append(mean(relu(dist - delta_v) ** 2))
    num = len(centroids)
    var_loss = mean(var_terms)
    dist_loss = 0.0
    if num > 1:
        # Push every ordered pair of centroids to 2 * delta_d apart  [eq. 4]
        for a in range(num):
            for b in range(num):
                if a != b:
                    gap = norm(centroids[a] - centroids[b])
                    dist_loss += relu(2 * delta_d - gap) ** 2
        dist_loss = dist_loss / (num * (num - 1))
    # Weighted sum  [eq. 5]
    return alpha * var_loss + beta * dist_loss

Losses §3.1 §4.1: total loss L = α·L_var + β·L_dist + segmentation loss, with α = β = 1 eq. 5 §4.1. The paper does not give a weight for the direction loss.

Evaluation §3.2: two families of metrics.

• Eulerian (dense, per pixel): IoU of rasterized maps eq. 6.
• Lagrangian (per curve): directed Chamfer distance between point samples on predicted and label curves, and its bidirectional sum eq. 7 eq. 8.
• Instance: average precision over ten recall levels 0.1 to 1.0, with a prediction counted as a true positive when its Chamfer distance to a label is under a threshold; predictions are ranked by semantic confidence eq. 9 tab. II.

Hyperparameters

Values marked "not stated" do not appear in the paper. Code values are from the official repository at code_commit and are the defaults in the training script, not stated in the paper. The paper's implementation section is §4.1.

Results

Author-reported. Setup: nuScenes §4.1; surrounding cameras only unless noted; baselines IPM, IPM(B), IPM(CB), Lift-Splat-Shoot and VPN use the same encoder and decoder as HDMapNet and differ only in the view transformation §4.2. The paper does not state the split, model size or hardware.

Table I: semantic map segmentation. IoU in %, CD in m; lower CD is better tab. I.

Column key: Div = divider, Ped = pedestrian crossing, Bnd = boundary; All = all classes. The source table marks IPM with an asterisk: perspective-view labels projected from 2D HD maps tab. I.

Table II: instance map detection. AP at Chamfer-distance thresholds 0.2, 0.5 and 1.0 (units not stated in the compiled paper), and mAP, the mean of the three tab. II.

Table III: HDMapNet(Fusion) IoU by weather: night 39.3, rainy 38.7, normal 44.9 tab. III. The paper reports performance drops in extreme weather but says overall performance is still reasonable §4.3.

Table IV: HDMapNet(Surr) IoU with temporal fusion of 1, 2 and 4 frames: 32.9, 35.8, 36.4 tab. IV. Fusion is by max pooling of feature maps aligned with ego poses §4.3.

Headline claims in the text: the fusion model outperforms both single-sensor models by a large margin §4.3; the abstract says the fusion model outperforms existing methods "by more than 50% in all metrics", and the introduction gives gains of 12.1 IoU and 13.1 mAP §1. Both claims are checked in Pitfalls.

Paper vs code

Official code: Tsinghua-MARS-Lab/HDMapNet at 7911227d45edb0226422083bc4bc452dc18e6399, listed by the README with the four paper authors and linking the paper. License GPL-3.0 (copyleft): nothing is quoted from it; all code is cited by path and line and described in our own words.

• Direction branch reuses the embedding upsampler. The direction head is computed with up1_embedded instead of its own up1_direction block, so the two branches share that block's weights and the up1_direction block is unused base.py base.py. Paper says separate heads §3.1.
• Camera aggregation: max, not average. The image BEV features of the N cameras are combined with an element-wise max homography.py. Paper says average §3.1.
• Distance loss margin is δ_d, not 2δ_d, and is halved. The code hinge is relu(delta_d - dist) ** 2 with an extra division by 2 over ordered pairs loss.py. Paper's eq. 4 uses margin 2δ_d over ordered pairs with normalization C(C−1) eq. 4. So the code's distance term is smaller than the equation by a factor of 2 in addition to using a margin half as large.
• Semantic loss is per-class sigmoid BCE with pos_weight 2.13. Targets are one channel per class plus background, and the loss is BCEWithLogits(pos_weight=2.13) loss.py train.py dataset.py. Paper says cross-entropy §4.1. The vectorization applies a softmax across channels and then argmax vectorize.py evaluate.py.
• Direction targets are 0.5 and 0.5, not 1 and 1. Forward and backward one-hot labels are summed and normalized over channels, so the two lane directions share mass 0.5 each when they differ dataset.py. Paper says two indices set to 1 §3.1. The direction loss is a BCE on softmax outputs, masked to lane pixels, and weighted 0.2 in the total train.py train.py; the paper gives neither the loss form nor the weight.
• Direction bins. The code uses 36 direction classes at 10° plus a no-direction class train.py hdmapnet.py. Paper leaves N_d unstated §3.1.
• Evaluation thresholds differ. The code sets the AP thresholds to 2, 4 and 6 evaluate_json.py and multiplies coordinates by 0.15 m per cell before comparing AP.py evaluate_json.py. That gives thresholds of 2, 4 and 6 m. The paper's table uses 0.2, 0.5 and 1.0 with no units tab. II. Confirm before comparing numbers.
• Chamfer aggregation. The code pools points across the dataset: the directed sums are divided by the total number of points of both sets evaluate_json.py. The table caption says CD is the average of CD_P and CD_L tab. I. The reported values match neither the paper's sum of two directed terms eq. 8 nor the arithmetic mean of CD_P and CD_L (for HDMapNet(Surr), the mean of 0.741 and 1.267 is 1.004, against the reported 0.834).
• Connection. The code grows lines from a random start point in both directions connect.py, so runs are not deterministic unless the seed is fixed. Step size is 7 cells, about 1.05 m at 0.15 m per cell vectorize.py.
• Dataset class names. The code maps "road_divider" and "lane_divider" to class 0, "ped_crossing" to 1, and "contours" to 2 const.py. The paper calls class 2 lane boundary.
• Training details. Epochs (30), batch size (4), weight decay (1e-7), StepLR, gradient clipping and the BEV resolution appear only in the code train.py.

Pitfalls

• Unstated geometry. The paper gives no BEV extent, no cell size and no image size. The compiled text dropped the commented-out values (for example, a 0.15 m grid with a 200 × 400 BEV) §4.1. Use the code's values, listed in the hyperparameter table, and say which you used.
• Ambiguous view transform. Eq. 1 gives a separate MLP per camera over all perspective pixels eq. 1. The text says the MLP is shared channel-wise §4.1, and the code applies the same two linear layers over the spatial grid for each channel hdmapnet.py. Width, depth and output size are not stated in the paper; the code's output is 40 × 80 cells per camera hdmapnet.py.
• Loss margins. Implement eq. 4 as written only if you also change the code's margin; the official code is the source of the trained model and uses δ_d with a halved term. Pick one and document it.
• Projection order. The paper describes transform to the camera frame then projection to ego BEV. The code maps each camera's perspective features into a 40 × 80 top-down grid with per-camera MLPs and then uses an inverse-perspective warp with extrinsics, pooled by max hdmapnet.py hdmapnet.py homography.py.
• Sensor fusion is a concatenation. Image BEV and LiDAR BEV are concatenated along channels after upsampling; no fusion weights or gates are described hdmapnet.py. The source tree contains a sensor-fusion table that is not part of the compiled paper (tables/sensor_fusion.tex); its scales differ from Table II and its camera-only mAP column is 0, so do not use it.
• Headline gains are not uniform. The abstract's "more than 50% in all metrics" does not hold for every metric. All-class IoU is 44.5 for fusion against 32.4 for IPM(CB) and 32.9 for HDMapNet(Surr), gains of about 37% and 35% tab. I. The intro's +12.1 IoU and +13.1 mAP §1 are measured against IPM(CB) for IoU and against VPN (17.5) for mAP; against IPM(CB), the best mAP baseline (19.7), the gain is 10.9 tab. II.
• Algorithms are not in the compiled paper. The vectorization algorithms are in a separate source file that the paper does not include §3.1; this card cites the code for them instead.
• Chamfer caption. The CD column description is loose (see Paper vs code): CD is not the arithmetic mean of CD_P and CD_L.
• Equation 7 typography. The source writes the normalization as 1/S_1 with a set macro; read it as 1/|S_1| eq. 7.
• Evaluation is on nuScenes only. The split is not stated; code defaults to v1.0-mini and the mini validation split evaluate_json.py.
• Finetuning mode. The code can load a checkpoint and train only layers whose names contain bevencode.up or bevencode.layer3 train.py; the paper does not mention it.
• Cost. The BEV decoder runs a ResNet at 200 × 400 resolution with three heads, and the view transform is a dense MLP over all perspective pixels for each camera hdmapnet.py base.py; the paper gives no runtime or memory figures.
• Instance evaluation cost. Chamfer matching is computed pairwise for every prediction and label per class and threshold on GPU AP.py; this is slow on large sets.

Lineage

• Builds on: the view transform and camera-to-BEV idea from Lift-Splat-Shoot (named in §4.2; no arXiv link in the paper), PointPillars (named in §3.1), the discriminative instance loss of Brabandere et al. (named in §3.1), and FCN decoders (named in §3.1). No cards for these papers exist yet.
• Compared against: IPM, IPM(B), IPM(CB), Lift-Splat-Shoot and VPN §4.2 tab. I tab. II.
• Superseded by: not stated in the paper; nothing in this wiki yet supersedes it.