Paper · 2024-03-25 · cs.CV
RCBEVDet: Radar-camera Fusion in Bird's Eye View for 3D Object Detection
RCBEVDet fuses multi-view cameras with millimeter-wave radar on a bird's-eye-view grid to detect 3D objects. A dual-stream radar encoder and an RCS-aware scattering step build the radar BEV map, and deformable cross-attention aligns it with the camera BEV map before fusion. Author-reported: 56.8 NDS and 45.3 mAP on nuScenes val with ResNet-50, at 21.3 FPS.
Bird's-eye-view (BEV) representationMultimodal fusionCross-attentionObject detection3d-object-detectionradar-camera-fusionbird-eye-viewautonomous-driving
arXivPDFCodeMarkdown16 min read
AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Self-driving cars need to know where other objects sit in 3D. Cameras show colour and texture well, but they judge distance poorly and struggle in bad weather or at night. Radar works in weather and light and measures distance and speed well, but its returns are sparse and say little about what an object looks like.
RCBEVDet combines the two. It turns the camera images and the radar returns into the same top-down map of the scene, the bird's-eye view, and merges them there. Two ideas make the radar side work better. First, each radar return is spread over nearby map cells, with its spread set by how strongly the object reflects radar, a rough clue to its size. Second, an attention step lets the camera and radar maps shift slightly to line up, because radar positions are often a little off.
The result is a detector that uses radar to complement cameras while still running in real time.
TL;DR
RCBEVDet is a camera-plus-radar 3D object detector that works in bird's-eye view (BEV). Its radar branch has a point-based and a transformer-based encoder that exchange features, and an RCS-aware step that spreads each radar return over nearby BEV cells according to its radar cross-section. A deformable cross-attention module aligns the radar and camera BEV maps before a channel-and-spatial fusion. Author-reported: 56.8 NDS and 45.3 mAP on nuScenes val with ResNet-50 at 21.3 FPS. The official code is academic-only, so this card is based on the paper text alone.
Problem
- Cameras give rich appearance but cannot measure depth precisely, and they fail in adverse weather or low light §1.
- Radar is largely unaffected by weather and lighting and measures distance and velocity, but its data are sparse and carry little semantic information §1 §2.2.
- Earlier radar-camera fusion methods align the two sensors poorly, using element-wise concatenation, summation or SE-style weighting, and cannot handle the spatial misalignment between camera and radar inputs §1.
- Earlier radar-camera methods reuse LiDAR pillar encoders for radar, which the authors call sub-optimal §1.
- Radar points often carry azimuth errors, so radar features can fall into neighbouring BEV cells §3.2.
- The radar BEV map produced by plain voxel scattering is sparse, and deeper BEV encoders blur small objects into the background §3.1.
Method
Overview §3 fig. 2:
- Image branch. Multi-view images go through an image encoder and a view transform into an image BEV feature. The branch adapts BEVDepth: it accumulates BEV features from previous frames, passes them through an extra encoder of two residual blocks following BEVDet4D, and uses BEVPoolv2 for the view transform §4.1.
- Radar branch. Aligned radar point clouds are accumulated over several sweeps, each point carrying RCS and Doppler speed as features §3 §4.1. The RadarBEVNet encodes them into a radar BEV feature §3.1.
- Fusion. The cross-attention multi-layer fusion module (CAMF) aligns the two BEV maps and fuses them §3.2 fig. 6.
- Detection. The fused BEV feature feeds the 3D detection task. The detection head is not described §3.
RadarBEVNet, dual-stream radar backbone §3.1 fig. 3 fig. 4:
- Point stream. S blocks of a PointNet-like plain structure. Each block applies an MLP to the per-point features, max-pools over all points, and concatenates the pooled global feature back onto each point eq. 1.
- Transformer stream. S standard transformer blocks (attention, feed-forward network, normalization). Its attention is distance-modulated (DMSA): a Gaussian-like weight, with a learnable width, down-weights far-away point pairs, so early training favours neighbours eq. 2 eq. 3. Multi-head DMSA gives each head its own learnable factor eq. 4.
- Injection (point stream reads transformer stream). Point features are the queries and transformer features are keys and values in multi-head cross-attention. The result is added back to the point features through a learnable scale γ eq. 5 fig. 4.
- Extraction (transformer stream reads point stream). Transformer features are the queries and point features are keys and values in cross-attention, followed by a feed-forward network eq. 6 fig. 4.
- Injection and extraction are applied in every block of both streams §3.1.
RCS-aware BEV encoder §3.1 fig. 5:
- Plain voxel scattering writes each radar point into one BEV pixel. The RCS-aware encoder instead scatters the point's feature into many pixels, within a radius set by the point's RCS and its coordinate norm §3.1.
- Where several points land on one pixel, their features are summed §3.1.
- Each point also gets a Gaussian-like weight map whose width depends on its RCS and coordinate. The final weight map keeps the maximum over all points eq. 7.
- The scattered features and the weight map are concatenated and passed through an MLP eq. 8. The result is concatenated with the ordinary BEV feature and fed to a SECOND-style BEV encoder §3.1.
Cross-attention multi-layer fusion, CAMF §3.2 fig. 6:
- Learnable position embeddings are added to the camera and radar BEV maps §3.2.
- Radar BEV feature gives queries and reference points, and the camera BEV feature gives keys and values, in multi-head deformable cross-attention eq. 9. The roles are then swapped to update the radar map from the camera map eq. 10.
- The authors state that deformable attention lowers the cost from O(H²W²C) to O(HWC²K) §3.2.
- Channel and spatial fusion: the two aligned maps are concatenated, passed through one CBR block with a residual connection, then through three more CBR blocks. A CBR block is a 3×3 convolution, batch normalization and ReLU §3.2.
Training §4.1: two stages. The camera stream is trained first, following the standard BEVDepth implementation. The fusion model is then trained with the camera-stream weights frozen. Training runs 12 epochs with AdamW. The authors apply image and radar augmentations and class-balanced sampling (CBGS).
Equations, reproduced in full from the paper:
f = \text{Concat}[\text{MLP}(f),\text{MaxPool}(\text{MLP}(f))] eq. 1
G_{i,j} = \exp(-D_{i,j}^2/\sigma^2), \quad \sigma \text{ learnable} eq. 2
\text{DMSA}(Q, K, V) = \text{Softmax}\left(\frac{QK^{\top}}{\sqrt{d}}+\log G\right)V = \text{Softmax}\left(\frac{QK^{\top}}{\sqrt{d}}-\frac{1}{\sigma^2}D^2\right)V eq. 3
\text{head}_i = \text{Softmax}\left(\frac{Q_iK_i^{\top}}{\sqrt{d_i}}-\beta_iD^2\right)V_i, \quad \text{MultiHeadDMSA} = \text{Concat}[\text{head}_1,\dots,\text{head}_H] eq. 4
f_p^i = f_p^i + \gamma\times \text{CrossAttention}(LN(f_p^i), LN(f_t^i)) eq. 5
f_t^i = \text{FFN}(f_t^i + \text{CrossAttention}(LN(f_t^i), LN(f_p^i))) eq. 6
G_{x,y}=\exp\left(-\frac{(c_x-x)^2+(c_y-y)^2}{\frac{1}{3}(c_x^2+c_y^2)\times v_{RCS}}\right) eq. 7
f_{RCS}'=\text{MLP}(\text{Concat}(f_{RCS},G_{RCS})) eq. 8
\text{DeformAttn}(z_{q_r}, p_{q_r}, F_c) = \sum_{m=1}^{M} W_m\left[\sum_{k=1}^{K} A_{mqk} \cdot W'_m F_c(p_{q_r}+\Delta p_{mqk})\right] eq. 9
F_c \leftarrow \text{DeformAttn}(z_{q_r}, p_{q_r}, F_c), \qquad F_r \leftarrow \text{DeformAttn}(z_{q_c}, p_{q_c}, F_r) eq. 10
Symbols as defined in the paper: f_p and f_t are point-stream and transformer-stream features at block i; c and (x, y) are a radar point's BEV coordinate and a pixel coordinate; v_RCS is the RCS value; F_c and F_r are camera and radar BEV maps; Δp are sampling offsets; A are attention weights §3.1 §3.2.
The pseudocode below is our paraphrase of the method. Where the paper leaves a step open, the line says "not stated".
# Dual-stream radar backbone [§3.1] [fig. 3] [fig. 4]
# Input: radar points (N, d_in), with RCS and Doppler features [§4.1]
# Output: point features and transformer features after S blocks
def dual_stream_radar(points, S):
# Input projections of the two streams are not stated [§3.1]
point_feats = point_input(points) # (N, C)
token_feats = token_input(points) # (N, C)
for block in range(S):
# Point stream: per-point MLP plus global max-pool [eq. 1]
point_feats = point_block(point_feats) # (N, C)
# Transformer stream: attention with distance modulation [eq. 3-4]
token_feats = transformer_block(token_feats, points) # (N, C)
# Injection: point features query the transformer features [eq. 5]
# Whether extraction sees these updated features is not stated [§3.1]
point_feats = point_feats + gamma * cross_attn(
layer_norm(point_feats), layer_norm(token_feats))
# Extraction: transformer features query the point features [eq. 6]
token_feats = ffn(token_feats + cross_attn(
layer_norm(token_feats), layer_norm(point_feats)))
return point_feats, token_feats# RCS-aware scattering into the radar BEV grid [§3.1] [eq. 7] [eq. 8]
# Input: BEV coordinates (N, 2), features (N, C), RCS (N,)
# Output: radar BEV feature (H, W, C) after the MLP
def rcs_scatter(coords, feats, rcs, H, W):
# Summed features and the maximum weight start at zero [§3.1]
bev_feats = zeros(H, W, C)
weight_map = zeros(H, W)
for i in range(N):
# Radius from coordinate norm times RCS; units not stated [§3.1]
norm_sq = coords[i, 0] ** 2 + coords[i, 1] ** 2
radius = norm_sq * rcs[i]
for each pixel p with distance(p, coords[i]) < radius:
# Sum-pool the point feature into every pixel in range [§3.1]
bev_feats[p] += feats[i]
# Gaussian-like weight; keep the maximum over points [eq. 7]
dist_sq = distance(p, coords[i]) ** 2
weight = exp(-dist_sq / (norm_sq * rcs[i] / 3))
weight_map[p] = max(weight_map[p], weight)
# Concatenate features with weights, then an MLP [eq. 8]
fused = mlp(concat(bev_feats, weight_map))
# Concatenate with the ordinary BEV feature for the BEV encoder [§3.1]
return fused# Cross-attention multi-layer fusion [§3.2] [eq. 9] [eq. 10] [fig. 6]
# Input: camera BEV (C_c, H, W), radar BEV (C_r, H, W)
# Output: fused BEV (C, H, W)
def camf(cam_bev, radar_bev):
# Learnable position embeddings added to both maps [§3.2]
cam_bev = cam_bev + cam_pos_embed
radar_bev = radar_bev + radar_pos_embed
# Query and reference-point projections are not stated [§3.2]
radar_queries, radar_refs = query_head(radar_bev)
cam_queries, cam_refs = query_head(cam_bev)
# Radar gives queries; camera gives keys and values [eq. 9]
cam_aligned = deform_attn(radar_queries, radar_refs, cam_bev)
# Swap roles to update the radar map [eq. 10]
radar_aligned = deform_attn(cam_queries, cam_refs, radar_bev)
# Concatenate the two aligned maps [§3.2]
fused = concat(cam_aligned, radar_aligned)
# One CBR block with a residual connection [§3.2]
fused = fused + cbr(fused)
# Three more CBR blocks [§3.2]
for _ in range(3):
fused = cbr(fused)
return fusedHyperparameters
Values stated in the paper. The camera stream inherits its settings from BEVDepth, which this paper does not restate.
| Name | Value | Where | Notes |
|---|---|---|---|
| Training schedule | 12 epochs | §4.1 | |
| Optimizer | AdamW | §4.1 | learning rate, weight decay, warm-up and schedule: not stated |
| Training stages | 2: camera stream first, then fusion with camera weights frozen | §4.1 | |
| Camera stream | BEVDepth with temporal BEV accumulation | §4.1 | BEVDepth's own settings are not restated here |
| Extra BEV encoder | 2 residual blocks | §4.1 | follows BEVDet4D, applied before temporal fusion |
| Previous frames | number not stated | §4.1 | |
| View transform | BEVPoolv2 | §4.1 | |
| Radar input | multi-sweep points; features: RCS and Doppler speed | §4.1 | number of sweeps not stated |
| Radar BEV encoder | SECOND-style encoder after RCS-aware scattering | §3.1 | encoder depth and widths not stated |
| Dual-stream stages S | 3 | §4.1 | §3.1 also uses S for the block count; per-backbone value not stated |
| Attention heads H (DMSA) | not stated | §3.1 eq. 4 | each head has its own β |
| DMSA width σ or β | learnable; initial value not stated | §3.1 eq. 2-4 | β = 0 gives vanilla self-attention |
| Injection/extraction scale γ | learnable; initial value not stated | §3.1 eq. 5 | |
| Sampled keys K, heads M (deformable attention) | not stated | §3.2 eq. 9 | |
| CBR blocks | 1 with residual, then 3; 3×3 conv, BN, ReLU | §3.2 | channel widths not stated |
| Data augmentation | image and radar augmentations; types not stated | §4.1 | |
| Class-balanced sampling | CBGS | §4.1 | |
| Ablation baseline input | 256×704 images, 128×128 BEV grid | §4.3 | BEV extent in metres not stated |
| Backbones evaluated | ResNet-18, ResNet-50, DLA-34, Swin-T (val); V2-99 and ConvNeXt-B (test) | tab. 1 tab. 2 | |
| Temporal supervision | extra per-frame losses, following HoP; weights not stated | §4.3 | |
| Inference timing | RTX 3090, batch size 1, FP16 | §4.1 | |
| Batch size, learning rate | not stated | §4.1 |
Results
Author-reported. Setups:
- nuScenes val and test sets, multi-view cameras and radar, with the comparison models listed in tab. 1 and tab. 2 §4.2.
- VoD validation set, camera and radar, with entire-area and region-of-interest AP tab. 3 §4.2.
- Ablations run on nuScenes val with ResNet-50, 256×704 images and a 128×128 BEV grid §4.3.
- Speed measured on one RTX 3090, batch size 1, FP16 §4.1.
Headline numbers:
| Setting | Metric | Baseline | This paper | Source |
|---|---|---|---|---|
| nuScenes val, ResNet-50, 256×704 | NDS / mAP | CRN 56.0 / 49.0; SOLOFusion (camera only) 53.4 / 42.7 | 56.8 / 45.3 | tab. 1 |
| nuScenes val, ResNet-50 | FPS | CRN 20.4 | 21.3 | tab. 1 |
| nuScenes test, V2-99 | NDS / mAP | BEVDepth* (camera, V2-99) 60.5 / 51.5; CRN ConvNeXt-B 62.4 / 57.5 | 63.9 / 55.0 | tab. 2 |
| VoD val, region of interest | mAP | RCFusion 69.23 | 69.80 | tab. 3 |
| VoD val, entire annotated area | mAP | RCFusion 49.65 | 49.99 | tab. 3 |
| nuScenes val, ablation | NDS / mAP | BEVDepth + temporal 51.9 / 40.5 | 56.8 / 45.3 | tab. 4 |
Full nuScenes val comparison, as reported. Model names are the paper's; the "(ours)" rows are RCBEVDet. "Resolution" is the input image size. FPS is the authors' measurement:
| Model | Sensors | Backbone | Resolution | NDS | mAP | mATE (translation) | mASE (scale) | mAOE (orientation) | mAVE (velocity) | mAAE (attribute) | FPS |
|---|---|---|---|---|---|---|---|---|---|---|---|
| CenterFusion | C+R | DLA-34 | 448×800 | 45.3 | 33.2 | 0.649 | 0.263 | 0.535 | 0.540 | 0.142 | – |
| CRAFT | C+R | DLA-34 | 448×800 | 51.7 | 41.1 | 0.494 | 0.276 | 0.454 | 0.486 | 0.176 | 4.1 |
| RCBEVDet (ours) | C+R | DLA-34 | 448×800 | 56.3 | 45.3 | 0.492 | 0.269 | 0.449 | 0.230 | 0.188 | 4.7 |
| RCBEV4d | C+R | Swin-T | 256×704 | 49.7 | 38.1 | 0.526 | 0.272 | 0.445 | 0.465 | 0.185 | – |
| RCBEVDet (ours) | C+R | Swin-T | 256×704 | 56.2 | 49.6 | 0.496 | 0.271 | 0.418 | 0.239 | 0.179 | 18.2 |
| CRN | C+R | R18 | 256×704 | 54.3 | 44.8 | 0.518 | 0.283 | 0.552 | 0.279 | 0.180 | 27.9 |
| RCBEVDet (ours) | C+R | R18 | 256×704 | 54.8 | 42.9 | 0.502 | 0.291 | 0.432 | 0.210 | 0.178 | 28.3 |
| BEVDet | C | R50 | 256×704 | 39.2 | 31.2 | 0.691 | 0.272 | 0.523 | 0.909 | 0.247 | 15.6 |
| BEVDepth | C | R50 | 256×704 | 47.5 | 35.1 | 0.639 | 0.267 | 0.479 | 0.428 | 0.198 | 11.6 |
| SOLOFusion | C | R50 | 256×704 | 53.4 | 42.7 | 0.567 | 0.274 | 0.411 | 0.252 | 0.188 | 11.4 |
| StreamPETR | C | R50 | 256×704 | 54.0 | 43.2 | 0.581 | 0.272 | 0.413 | 0.295 | 0.195 | 27.1 |
| CRN | C+R | R50 | 256×704 | 56.0 | 49.0 | 0.487 | 0.277 | 0.542 | 0.344 | 0.197 | 20.4 |
| RCBEVDet (ours) | C+R | R50 | 256×704 | 56.8 | 45.3 | 0.486 | 0.285 | 0.404 | 0.220 | 0.192 | 21.3 |
tab. 1 Source: the paper's nuScenes val table; bold best-value marking omitted.
nuScenes test comparison, as reported. The asterisk marks the camera model in RCBEVDet, implemented from the released BEVDepth code. The test table has no FPS column:
| Model | Sensors | Backbone | NDS | mAP | mATE (translation) | mASE (scale) | mAOE (orientation) | mAVE (velocity) | mAAE (attribute) |
|---|---|---|---|---|---|---|---|---|---|
| KPConvPillars | R | Pillars | 13.9 | 4.9 | 0.823 | 0.428 | 0.607 | 2.081 | 1.000 |
| CenterFusion | C+R | DLA-34 | 44.9 | 32.6 | 0.631 | 0.261 | 0.516 | 0.614 | 0.115 |
| RCBEV | C+R | Swin-T | 48.6 | 40.6 | 0.484 | 0.257 | 0.587 | 0.702 | 0.140 |
| MVFusion | C+R | V2-99 | 51.7 | 45.3 | 0.569 | 0.246 | 0.379 | 0.781 | 0.128 |
| CRAFT | C+R | DLA-34 | 52.3 | 41.1 | 0.467 | 0.268 | 0.456 | 0.519 | 0.114 |
| BEVFormer | C | V2-99 | 56.9 | 48.1 | 0.582 | 0.256 | 0.375 | 0.378 | 0.126 |
| PETRv2 | C | V2-99 | 58.2 | 49.0 | 0.561 | 0.243 | 0.361 | 0.343 | 0.120 |
| BEVDepth* | C | V2-99 | 60.5 | 51.5 | 0.446 | 0.242 | 0.377 | 0.324 | 0.135 |
| BEVDepth | C | ConvNeXt-B | 60.9 | 52.0 | 0.445 | 0.243 | 0.352 | 0.347 | 0.127 |
| BEVStereo | C | V2-99 | 61.0 | 52.5 | 0.431 | 0.246 | 0.358 | 0.357 | 0.138 |
| SOLOFusion | C | ConvNeXt-B | 61.9 | 54.0 | 0.453 | 0.257 | 0.376 | 0.276 | 0.148 |
| CRN | C+R | ConvNeXt-B | 62.4 | 57.5 | 0.416 | 0.264 | 0.456 | 0.365 | 0.130 |
| SparseBEV | C | V2-99 | 63.6 | 55.6 | 0.485 | 0.244 | 0.332 | 0.246 | 0.117 |
| StreamPETR | C | V2-99 | 63.6 | 55.0 | 0.493 | 0.241 | 0.343 | 0.243 | 0.123 |
| RCBEVDet (ours) | C+R | V2-99 | 63.9 | 55.0 | 0.390 | 0.234 | 0.362 | 0.259 | 0.113 |
tab. 2 Source: the paper's nuScenes test table.
VoD validation, as reported. Matching IoU is 0.5 for the car class and 0.25 for the other two classes. Region of interest: a corridor near the ego vehicle tab. 3:
| Model | Sensors | Car | Pedestrian | Cyclist | mAP | Car (ROI) | Pedestrian (ROI) | Cyclist (ROI) | mAP (ROI) |
|---|---|---|---|---|---|---|---|---|---|
| PointPillar | R | 37.06 | 35.04 | 63.44 | 45.18 | 70.15 | 47.22 | 85.07 | 67.48 |
| RadarPillarNet | R | 39.30 | 35.10 | 63.63 | 46.01 | 71.65 | 42.80 | 83.14 | 65.86 |
| RCFusion | C+R | 41.70 | 38.95 | 68.31 | 49.65 | 71.87 | 47.50 | 88.33 | 69.23 |
| RCBEVDet (ours) | C+R | 40.63 | 38.86 | 70.48 | 49.99 | 72.48 | 49.89 | 87.01 | 69.80 |
tab. 3 Source: the paper's VoD validation table. The first four AP columns are the entire annotated area, the last four the region of interest.
Ablation of main components, nuScenes val, cumulative tab. 4:
| Model configuration | Input | NDS | mAP |
|---|---|---|---|
| BEVDepth | C | 47.5 | 35.1 |
| + temporal | C | 51.9 | 40.5 |
| + PointPillar and BEVFusion | C+R | 53.6 | 42.3 |
| + RadarBEVNet | C+R | 55.7 | 45.3 |
| + CAMF | C+R | 56.4 | 45.6 |
| + temporal supervision | C+R | 56.8 | 45.3 |
Ablation of the radar backbone, nuScenes val tab. 5. The paper's table does not say which fusion module the rows use (see Pitfalls):
| Radar encoder variant | NDS | mAP |
|---|---|---|
| Point stream only | 54.3 | 42.6 |
| + RCS-aware scatter step | 55.7 | 44.5 |
| + transformer stream | 55.8 | 44.8 |
| + injection and extraction | 56.4 | 45.6 |
Ablation of the fusion module, nuScenes val tab. 6:
| Fusion method | NDS | mAP |
|---|---|---|
| BEVFusion fusion | 55.7 | 45.3 |
| + deformable cross-attention | 56.1 | 45.5 |
| + channel and spatial fusion | 56.4 | 45.6 |
Robustness: Car-class mAP when camera or radar inputs are dropped in training and testing, following CRN tab. 7 §4.4. Column headings are the paper's: the "0" column is merged across both rows of each fusion method and applies to both. The paper's text calls the radar-row columns "radar sensor failure cases" (see Pitfalls):
| Model | Input | Dropped | 0 | 1 | 3 | All |
|---|---|---|---|---|---|---|
| BEVDepth | C | C | 49.4 | 41.1 | 24.2 | 0 |
| CenterPoint | R | R | 30.6 | 25.3 | 14.9 | 0 |
| BEVFusion | C+R | C | 63.9 | 58.5 | 45.7 | 14.3 |
| BEVFusion | C+R | R | 63.9 | 59.9 | 50.9 | 34.4 |
| CRN | C+R | C | 68.8 | 62.4 | 48.9 | 12.8 |
| CRN | C+R | R | 68.8 | 64.3 | 57.0 | 43.8 |
| RCBEVDet (ours) | C+R | C | 72.5 | 66.9 | 53.5 | 16.5 |
| RCBEVDet (ours) | C+R | R | 72.5 | 71.6 | 66.1 | 62.1 |
Other results named in the text §4.2 §4.3:
- Versus the camera-only SOLOFusion on nuScenes val with ResNet-50, the mAVE is reported as 14.7% lower, and versus CRN, 37.5% lower (see Pitfalls for the recomputed figures).
- On the test set, RCBEVDet improves the BEVDepth baseline by 3.4 NDS and 3.5 mAP.
- Temporal accumulation adds 4.4 NDS and 5.4 mAP to BEVDepth; radar with RadarBEVNet and CAMF adds the rest of the ablation gains tab. 4.
- Temporal supervision adds 0.4 NDS and costs 0.3 mAP tab. 4.
Paper vs code
Official repository: VDIGPKU/RCBEVDet, linked from the abstract. Its README says the project is free for academic research only and needs authorization for commercial use, and gives a contact address for that. The repository has no LICENSE file. The code is distributed as a zip archive in the repository, at commit 15c83cc, and the README also points to model weights on Google Drive. Because this site is commercial, we did not extract or read the code for this card. No paper-vs-code comparison has been made, and every hyperparameter marked "not stated" above stays unverified against the code.
Pitfalls
- Missing training details. The paper gives no learning rate, batch size, weight decay, schedule, augmentation types, sweep count, previous-frame count, BEV extent in metres, head design or channel widths. Implementers need these from the license-restricted code, from BEVDepth, or from a commercial permission. The code was not used here.
- β versus σ. §3.1 says β replaces 1/σ, but eq. 4 writes β_i D², which corresponds to replacing 1/σ² (eq. 3). Implement eq. 4 as written, and check the β = 0 case gives vanilla attention.
- S has two meanings. §3.1 uses S for the number of blocks in each stream. §4.1 calls S "the number of stages" and sets it to three. The paper does not say whether the two streams share it or how blocks map to stages.
- Units in RCS scattering. §3.1 defines the radius in pixel distance, but the expression uses the 3D coordinate c, and eq. 7 uses c directly. The units of c and of v_RCS are not stated. RCS can be negative in common radar conventions, which would break both the radius and the denominator of eq. 7. The paper does not state how RCS is normalized.
- Constant 1/3 in eq. 7. The 1/3 is not explained. Keep it as written when reproducing the weights.
- Order inside a block. The paper does not say whether extraction uses point features before or after injection, or where the module sits relative to each block's own attention and feed-forward layers. Eq. 6's residual covers only the cross-attention term, while eq. 5 has a learnable scale γ with no stated initial value.
- CAMF details. The paper does not say whether the second deformable attention (eq. 10) reads the first update or the original camera map. It does not give the projection that produces queries and reference points, the position-embedding shape, K or M, or how the residual on the CBR block is wired.
- Table 5 base. The Table 5 baseline (point stream only, 54.3 NDS) differs from the Table 4 base (BEVFusion with PointPillar, 53.6 NDS). The fusion module used in the Table 5 rows is not stated. Its last row, 56.4 / 45.6, equals the "+ CAMF" row of Table 4.
- mAVE percentages. The text says mAVE is 14.7% lower than SOLOFusion and 37.5% lower than CRN. From Table 1, 0.252 → 0.220 is 12.7%, and 0.344 → 0.220 is 36.0%. We report the table values. The paper may have used unrounded numbers.
- "Better than all" claims. RCBEVDet leads on NDS and mAVE in many rows, but not on mAP everywhere. CRN has higher mAP at ResNet-50 and ResNet-18 on val tab. 1, and at ConvNeXt-B on test tab. 2. The "state of the art" claim is mainly on NDS and velocity error.
- FPS. The abstract's 21–28 FPS range matches the ResNet-50 (21.3) and ResNet-18 (28.3) rows of Table 1. The DLA-34 row runs at 4.7 FPS and the Swin-T row at 18.2 FPS.
- Test-set baseline. The text's 3.4 NDS and 3.5 mAP gains over BEVDepth match the V2-99 row marked BEVDepth*, not the ConvNeXt-B BEVDepth row.
- Robustness columns. The header says "# of view drops", but the text describes the radar-row columns as radar failure cases. The meaning of the "All" column and the 0 for single-modality rows is not stated. Treat the robustness numbers as reported, not fully defined.
- Radar alignment. The paper says the radar points are "aligned" to the camera frame but does not describe the procedure. Frame-to-frame alignment for temporal accumulation is also not described.
- VoD region of interest. The exact bounds of the region of interest are not stated in the paper.
- Temporal supervision lowers mAP slightly while raising NDS tab. 4. The final model includes it.
- Author-reported. All numbers come from the authors, on nuScenes and VoD. We have not checked them.
Lineage
- Builds on: BEVDepth (camera stream and training recipe) §4.1; BEVDet4D (extra BEV encoder) §4.1; BEVPoolv2 (view transform) §4.1; SECOND (BEV encoder after radar scattering) §3.1; PointNet-style point blocks §3.1; transformer blocks §3.1; deformable attention §3.2; GRIFNet and CRN (radar features RCS and Doppler, as in CRN) §4.1; HoP (temporal supervision) §4.3.
- Replaces: the PointPillars radar encoder used in earlier radar-camera fusion §1 §4.3.
- Compared against: CenterFusion, CRAFT, RCBEV4d, CRN, RCFusion, MVFusion, BEVFormer, PETRv2, BEVStereo, SOLOFusion, StreamPETR, SparseBEV, KPConvPillars, RadarPillarNet tab. 1 tab. 2 tab. 3.
- Superseded by: not known from wiki content.
Source and license
Adapted from “RCBEVDet: Radar-camera Fusion in Bird's Eye View for 3D Object Detection” by Zhiwei Lin, Zhe Liu, Zhongyu Xia, Xinhao Wang, Yongtao Wang, Shengxiang Qi, Yang Dong, Nan Dong, Le Zhang, Ce Zhu (arXiv:2403.16440), used under CC0 1.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/VDIGPKU/RCBEVDet at commit 15c83ccdd5a8 (Custom academic-research-only license, non-commercial (per README; no LICENSE file)). Any quoted code is shown under that license.