CenterFusion: Center-based Radar and Camera Fusion for 3D Object Detection

In plain words

Self-driving cars need to know where other vehicles are, how far away they are and how fast they move. A camera shows what things look like but is weak at judging distance. Radar measures distance and speed directly and works in bad weather, but its returns are sparse and their height is unreliable.

CenterFusion starts from a camera-only detector that finds each object by the point at its centre in the image. It then decides which radar returns belong to each object by checking which ones fall inside a box-shaped region in front of the camera, around that object's estimated position. The matched radar readings (distance and speed) are painted onto small image-shaped maps, and a second set of prediction heads uses those maps to refine each object's depth, rotation and velocity.

The point of the method is that a single camera frame gives poor speed estimates, while radar reads speed directly. Combining the two gives better velocity estimates on the same frame, without looking at earlier frames.

TL;DR

CenterFusion is a middle-fusion 3D detector for nuScenes. A CenterNet-style camera detector proposes objects; a frustum-based step links radar points to each object (after expanding each point to a pillar); the matched depth and velocity become three extra feature channels for second-stage heads. The authors report NDS 0.453 against 0.328 for the camera baseline on the validation split (both DLA), and 0.449 on the test split against 0.400 for the Hourglass CenterNet. The code is MIT-licensed and in the authors' repository, but several parameters differ from the paper; read Paper vs code before reproducing numbers.

Problem

Cameras and LiDAR cannot estimate object velocity without temporal information, and cameras are weak at depth §1. Radars use the Doppler effect to measure velocity directly, reach long range (up to about 200 m for automotive radars) and are robust to adverse weather §1. Their point clouds are much sparser than LiDAR, and the vertical value is often inaccurate or missing §1. The radial velocity a radar reports is not the object's full velocity vector unless the object moves along the line of sight [§3.1, fig. 2].

The naive fix, projecting each radar point into the image and assigning it to any 2D box it falls in, fails. Points do not map one-to-one to objects; one object can produce many points, and some points belong to no object. Wrong height places points outside their box, and occluded objects project to nearby image areas §4.2. The method assumes nuScenes-style data: calibrated radar and camera, and radar aggregated over three sweeps covering the past 0.25 s §3.1.

Method

CenterFusion keeps the CenterNet camera pipeline and adds a radar branch between its two stages §4. Primary heads on image features give preliminary 2D and 3D boxes §4.1. Radar points are associated to each object's centre by a frustum built from the preliminary box, after pillar expansion of the points §4.2. The associated depth and velocity become three feature channels concatenated to the image features §4.3. Secondary heads then recompute depth and rotation and predict velocity and attributes §4.3 fig. 1.

1. Image detection. A modified Deep Layer Aggregation (DLA) backbone feeds primary heads. Each head is a 3×3 convolution with 256 channels and a 1×1 convolution. The heads output the centre heatmap, 2D size, centre offset, 3D dimensions, depth and rotation §4.1 §3.2.
2. Radar points. Each point is (x, y, z, v_x, v_y) in the ego frame, with radial velocity compensated for ego motion. Three sweeps of the past 0.25 s are aggregated §3.1.
3. Pillar expansion. Each radar point becomes a fixed-size pillar. A point counts as inside a frustum when all or part of its pillar is inside §4.2 fig. 4.
4. Frustum association. For each preliminary object, build a frustum from its 2D box. Inside it, build a 3D region of interest (RoI) from the estimated depth, size and rotation. Radar points inside the RoI are candidates; the closest one is kept §4.2 fig. 3. Training uses the ground-truth 3D box. Testing uses the estimated box, enlarged in the radial direction by a parameter δ §4.2 §5.
5. Radar feature maps. For each associated radar point, write three channels over a region centred on the object's 2D box centre. The region scales with the box through a parameter α. The channels are depth, v_x and v_y. Where two regions overlap, the object with the smaller depth dominates §4.3 fig. 4.
6. Fusion and secondary heads. Concatenate the radar maps to the image features. Secondary heads (three 3×3 convolutions followed by a 1×1 convolution) recompute depth and rotation and predict velocity and attributes. Decoding takes depth, velocity, rotation and attributes from the secondary heads and the other box properties from the primary heads §4.3.

# Input:  image (3, H, W); radar points (N, 5) as (x, y, z, vx, vy)
# Output: 3D boxes with depth, rotation, velocity and attributes

def centerfusion(image, radar_points, calib):
    # Primary heads: heatmap, 2D size, offset, 3D size, depth, rotation  [§4.1]
    image_feats = backbone(image)                     # (C, H/R, W/R)
    primary = primary_heads(image_feats)

    # Expand each radar point to a fixed-size pillar  [§4.2] [fig. 4]
    pillars = expand_to_pillars(radar_points, calib)  # 0.2 x 0.2 x 1.5 m

    # Per object, keep the closest pillar inside its frustum RoI  [§4.2]
    # At test time the RoI uses the estimated box, enlarged by delta  [§5]
    matches = frustum_associate(primary, pillars, calib)

    # Paint depth, vx and vy over each object's region  [§4.3]
    radar_maps = render_radar_maps(matches, primary)  # (3, H/R, W/R)

    # Secondary heads see image and radar features together  [§4.3]
    fused = concat(image_feats, radar_maps)           # (C + 3, H/R, W/R)
    refined = secondary_heads(fused)

    # Decode: refined depth, rotation, velocity, attributes  [§4.3]
    return decode(primary, refined)

The primary heatmap is built from Gaussians around object centres, with the peak value of each object's Gaussian kept where objects overlap §3.2:

Y_{qc} = \max_{i} \exp\left(-\frac{(p_{i}-q)^2}{2\sigma_{i}^2}\right) eq. 1

The heatmap is trained with a focal loss. The paper writes it with two exponents, α and β, whose values are not stated §3.2:

L_{k} = \frac{1}{N} \sum_{xyc} \begin{cases} (1 - \hat{Y}_{xyc})^{\alpha} \log(\hat{Y}_{xyc}) & Y_{xyc}=1 \\ (1-Y_{xyc})^{\beta} (\hat{Y}_{xyc})^{\alpha} \log(1-\hat{Y}_{xyc}) & \text{otherwise} \end{cases} §3.2, unnumbered

Each radar channel is a box-shaped window around the object, with value f_i divided by a normalizing factor M_i. The printed equation writes the y-centre as c_y^i; read it as c_y^j, the object's centre §4.3, unnumbered:

F^{j}_{x,y,i} = \frac{1}{M_i} \begin{cases} f_i & |x-c_{x}^{j}| \le \alpha w^{j} \ \text{and}\ |y-c_{y}^{j}| \le \alpha h^{j} \\ 0 & \text{otherwise} \end{cases} §4.3, unnumbered

Hyperparameters

Values are as the paper states them. The code column is in Paper vs code.

Results

Author-reported. nuScenes benchmark; DLA backbone for CenterFusion and the CenterNet baseline on the validation split; test-split comparisons include the Hourglass CenterNet, MonoDIS, OFT and InfoFocus (a LiDAR method) §6 tab. 1. Hardware and training are in §5.

Table 1: nuScenes 3D detection tab. 1

The paper's tables mark the best value in each column in bold; the markup is dropped here. mATE, mASE, mAOE, mAVE and mAAE are average translation, scale, orientation, velocity and attribute errors. C, R, L mark camera, radar and LiDAR input.

Table 2: per-class mAP on nuScenes tab. 2

Table 3: ablation on nuScenes validation tab. 3. Baseline is CenterNet (DLA), NDS 0.328, mAP 0.306. Other rows are percentage changes against that baseline (relative for NDS, absolute points for mAP, relative for the error metrics as printed). PE is pillar expansion, FA frustum association, FT flip test.

Table 4: per-class ablation on nuScenes validation tab. 4. Baseline values are per-class mAP in points; other rows are changes in points.

Qualitative results in camera view and bird's-eye view are shown in fig. 5; the text reads them as better box fit, especially for distant vehicles, and better velocity vectors §6.

Headline numbers the authors state: NDS up more than 12% over the camera-based state of the art (abstract); on test, a 12.25% relative NDS increase over CenterNet and 16.9% over MonoDIS §6; on validation, NDS up 38.1% and velocity error down 62.1% against CenterNet with the same DLA backbone §6. On test, the Hourglass CenterNet has 1.2 points more mAP than CenterFusion §6.

Paper vs code

Code: official repository github.com/mrnabati/CenterFusion, first author's account, at commit 3bac5c9b26daf958456a7bbbae5eb5a00a76d98a. License MIT LICENSE. Model weights are not in the repository; the README links them, and its validation and test figures match the paper's headline numbers README.md. The README also states that the released CenterNet baseline is the original 140-epoch model with 30 further epochs for velocity and attribute heads README.md, which matches §5.

Discrepancies, each with both citations:

• Test-time RoI enlargement. Paper enlarges the RoI length by 20% in the radial direction at test time §5. The code sets frustumExpansionRatio to 0.0 inside the point-cloud block, and no flag overrides it [code: src/lib/opts.py#L417]. It is applied as dist_thresh += dist_thresh * ratio pointcloud.py, so the code applies no enlargement.
• RoI half-width formula. The paper gives no formula §4.2. The code computes max(z) - min(z) / 2.0 over the rotated box corners [code: src/lib/utils/pointcloud.py#L204]. Corners are centred on the origin pointcloud.py, so z ranges from −e to +e and this expression equals 1.5 e. The natural half-extent is (max − min)/2 = e. The same expression appears in the dataset path [code: src/lib/dataset/generic_dataset.py#L29-L32]. This looks like an operator precedence slip; the paper does not say which was intended.
• Pillar size. Paper: 0.2 × 0.2 × 1.5 m in (x, y, z) §5. Code: a (height, width, length) triple of 1.5, 0.2, 0.2 [code: src/lib/opts.py#L299-L300], set unconditionally inside the point-cloud block opts.py. Because that assignment comes after parsing, the --pillar_dims 1.0,0.2,0.2 flag in the training script has no effect train.sh.
• Radar feature values. Paper: the depth channel is the normalized object depth d §4.3. Code: the depth written is the matched radar point's own depth, divided by max_pc_dist when normalize_depth is set [code: src/lib/utils/pointcloud.py#L309-L314] opts.py. The velocity channels are not normalized pointcloud.py. The channels are camera-frame v_x and v_z, read from the radar point's x and z velocity components generic_dataset.py; the paper names them ego-frame v_x and v_y §4.3.
• Overlap rule. Paper: where heatmap regions overlap, the smaller depth wins §4.3. Training path: points are written from farthest to nearest, so the nearest point wins generic_dataset.py. Test path: each of the top-K detections is written in score order with plain assignment, so a later detection overwrites an earlier one; no depth comparison appears pointcloud.py.
• Region size α. Paper: region width and height are set by α, value not stated §4.3. Code: fixed at 0.3 in the options, with no command-line flag opts.py pointcloud.py.
• Batch size. Paper: 26 §5. Training script: 32 [code: experiments/train.sh#L16]; the option default is also 32 [code: src/lib/opts.py#L130-L131].
• Learning rate and schedule. Not stated in the paper §5. Training script: 2.5e-4 with a drop at epoch 50 train.sh. Option default: 1.25e-4, with the drop at 60 opts.py. Optimizer: Adam main.py.
• Input resolution. Paper: 800×450 §5. Code: default 448 × 800 (height × width) nuscenes.py.
• Random shift. Paper: from 0 to 20% of image size §5. Code: a Gaussian with σ equal to shift, clipped to ±2 × shift times the crop scale generic_dataset.py. The training script passes 0.1 train.sh. This is not a uniform draw from 0 to 20%.
• Focal exponents. Not stated in the paper §3.2. Code fixes them at 2 and 4 losses.py.
• Losses. Paper: focal loss for the heatmap, L1 for most regression heads, BCE for attributes §5. Code: L1 for depth and velocity [code: src/lib/model/losses.py#L222] losses.py; smooth L1 for rotation residuals losses.py. Loss weights default to 1 for most heads and 0.1 for box size [code: src/lib/opts.py#L238-L259].
• Radar sweeps. Paper: three sweeps §3.1. Training and test scripts pass 3 train.sh test.sh; the option default is 1 opts.py. The sweep time window was not checked against the paper's 0.25 s.
• Max radar distance. Not in the paper. Option default 100 m [code: src/lib/opts.py#L281]; scripts use 60 m train.sh. Points beyond it are dropped before association [code: src/lib/dataset/generic_dataset.py#L386-L388].
• Per-class distance thresholds. The code defines per-class thresholds opts.py and converts them to class indices opts.py, but nothing reads the result. This is dead configuration.

No discrepancy found: training-time association uses the ground-truth depth, size and rotation generic_dataset.py, as the paper states §4.2. The heatmap model is initialised from CenterNet and velocity and attribute heads are trained on top README.md §5.

Pitfalls

• Symbol clash. The paper uses α in two places: the focal-loss exponent §3.2 and the radar region size §4.3. Read them separately. The code uses different parameters for each: focal exponents 2 and 4, region ratio 0.3.
• Printed equation typo. The radar feature equation writes c_y^i where it should be c_y^j §4.3. See the note in the Method section.
• Test-time association depends on predicted depth. The frustum at test uses the estimated depth, not the ground truth §4.2. The paper says a larger RoI catches radar points of nearby objects. The δ value is a key choice, and the paper and the code disagree on it §5 [code: src/lib/opts.py#L417].
• Unstated values that change results. Learning rate, optimizer, loss weights, focal exponents, α and M_i are not stated in the paper §3.2 §4.3 §5. The code sets some of them, as listed above.
• Velocity compensation can fall back silently. If an image record lacks a velocity transform, the code uses the identity matrix [code: src/lib/dataset/generic_dataset.py#L179-L182], so radar velocities are not ego-compensated in that case. The paper requires compensation §3.1.
• Points outside the image are dropped. Radar points are projected and filtered to the image before any association [code: src/lib/dataset/generic_dataset.py#L396-L398]. Points outside the field of view cannot contribute.
• Paper arithmetic. The text says the pillar and frustum steps together give a relative 35.5% NDS gain §7; the table says +34.5% tab. 3. The text says flip testing adds 3.9% mAP §7; the table difference is 8.4 − 4.3 = 4.1 points tab. 3. The text calls 38.1% an "absolute gain" §6; it is relative. The table says +37.8% tab. 3, while the rounded NDS values (0.453 vs 0.328) give +38.1%.
• Test mAP is lower than the Hourglass baseline. On the test split, CenterFusion's mAP is 0.326 against 0.338 for CenterNet with Hourglass [tab. 1]. The NDS and velocity gains are the main claim.
• Dataset not included. nuScenes data is needed to train or evaluate. Its terms are not checked in this card.

Lineage

• Builds on: CenterNet, arXiv:1904.07850 (no card here) §3.2; the DLA backbone §4.1; focal loss, cited in §3.2.
• Compared against: OFT, MonoDIS, CenterNet (Hourglass and DLA) and the LiDAR method InfoFocus tab. 1. Radar-camera related work in §2.2: Chadwick et al. (radar projected to the image), Nabati et al. 2020 (radar for 3D proposals) and CRF-Net (radar as vertical lines). The paper also covers RadarNet, a radar-LiDAR method §2.2. None has a card here.
• Superseded by: not known from wiki content.