Paper · 2021-03-30 · cs.CV
Depth-conditioned Dynamic Message Propagation for Monocular 3D Object Detection
Monocular 3D car detection on KITTI. A depth map from an external estimator feeds a second ResNet-50. At Stages II and III, each image location gathers 9 learned samples, weighted by affinities and gates from multi-scale depth features (DDMP). A training-only depth-centre task helps. Author-reported: 12.78 AP3D (R40, moderate) on KITTI test, 6 FPS on a V100.
Dynamic convolutionDeformable convolutionGraph neural networkMulti-task learning3d-object-detectionmonoculardepthkittimessage-passing
arXivPDFCodeMarkdown18 min read
AI-compiled by claude-haiku-5-5 on 2026-11-22, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Monocular 3D detection means finding cars in a single photo and estimating where each one is in 3D, how big it is and which way it faces, without a LiDAR sensor. The method adds a second input: a depth estimate of the same photo, made by a separate model.
Plain fusion of colour and depth is weak, so the network lets each image location look at a small set of nearby locations. Where to look is learned, and so are the weights for what it reads, and both come from depth features at several scales. The result is a feature map that knows both what the scene looks like and how far things are.
An extra task, used only during training, asks the depth branch to predict where each object's centre is in 3D. That pushes the depth features to focus on objects rather than just surfaces. The authors report that on the KITTI car benchmark the method ranked first among monocular methods on the day of its submission.
TL;DR
- Monocular 3D car detector on KITTI that uses a depth map from an external depth model as a second input §4.
- Its core block, DDMP, samples 9 neighbours per position with learned offsets, then mixes their features with depth-predicted affinities and gates §3.3.
- A centre-regression head on the depth branch is used only in training and adds no inference cost §3.5.
- Author-reported: 12.78 AP3D moderate on the KITTI test set (R40), against 11.72 for D4LCN; 20.39 on val1 (R40) tab. 1 tab. 2.
- Two ResNet-50 trunks and an external depth model make it heavier than the single-image baselines; use it when the depth step is affordable.
Problem
Image-only monocular detectors struggle with two things. Object scale changes with distance, which plain convolutions handle badly. And images carry little depth cue for 3D reasoning §1. Pseudo-LiDAR methods convert estimated depth to a point cloud, but they inherit the error of the depth estimate and drop appearance information §1. D4LCN builds depth-dependent kernels from depth features, but its local dilated kernels cannot cover the whole object context under perspective and occlusion, and its 3D performance depends heavily on the depth map §1.
This paper aims to learn features that are both context-aware and depth-aware, while making the model less sensitive to depth-estimate errors §1.
Method
Overview. Two branches share the same input resolution (512 × 1760, horizontal flip only) §4. The image branch is a ResNet-50 for detection. The depth branch is a second ResNet-50 that takes the depth map app. A fig. 2. DDMP modules sit at image Stage II and Stage III, and each one fuses image features with depth features from the stages below it §3.2 §3.4. Standard 3D heads for centre, size and orientation are placed on top §3.2. The auxiliary depth head is used only in training §3.5 fig. 2.
Graph message passing
Every feature position is treated as a node. A message-passing step gathers messages from a sampled set of K neighbours and updates the node §3.1. The formulation is:
\mathbf{m}_i^{t+1} = M^t\big(A_{i,j}\{\mathbf{h}_1^{(t)},\dots,\mathbf{h}_K^{(t)}\},\mathbf{w}_j\big) = \sum_{j \in \mathcal{N}(i)} A_{i,j}\,\mathbf{h}_j^{(t)}\,\mathbf{w}_j eq. 1
\mathbf{h}_i^{t+1} = U^t(\mathbf{h}_i^{t},\mathbf{m}_i^{t+1}) = \sigma\big(\mathbf{h}_i^{t} + \alpha_i^m\,\mathbf{m}_i^{t+1}\big) eq. 2
Here A_{i,j} is the connection weight, \mathbf{w}_j a transformation for each sampled node, \alpha_i^m a learned message scale, and \sigma a non-linearity such as ReLU §3.1. The paper states T iterations; the instantiation uses T = 1 §3.4.
Depth-conditioned dynamic message propagation (DDMP)
- Image walk. Sampling points are moved by a learned offset per node. The walk is a linear map of the node feature, with \mathbf{W}_{i,j} and \mathbf{b}_{i,j} learned on image nodes §3.3 eq. 3:
\Delta\mathbf{d}_{i,j} = \mathbf{W}_{i,j}\,\mathbf{h}_i + \mathbf{b}_{i,j} eq. 3
Here D = 2 is the spatial dimension, and K is 3 \times 3 = 9 points §3.3 app. A.
-
Depth walk and hybrid filters. Depth features are not sampled at image positions; a second walk is made on the depth feature map §3.3. Depth nodes then produce an affinity matrix A^l_{i,j} and a weight \mathbf{w}^l_{i,j} for each stage l §3.3 eq. 5.
\{A_{i,j}^{l};\ \mathbf{w}_{i,j}^{l}\} = \tilde{\mathbf{W}}_{i,j}^{l}\,\tilde{\mathbf{v}}_{i,j}^{l} + \tilde{\mathbf{b}}_{i,j}^{l} eq. 5
-
Multi-scale message. Messages are summed over depth stages l with balance weights \beta_l §3.3:
\mathbf{m}_i^{t+1} = \sum_{l \in L} \sum_{j \in \mathcal{N}(i)} \beta_l\,A_{i,j}^{l}\,\mathbf{h}'^{\,l,(t)}_j\,\mathbf{w}_{i,j}^{l} eq. 4
and, with the bilinear sampler \delta applied to the walked positions §3.3:
\mathbf{m}_i^{t+1} = \sum_{l \in L} \sum_{j \in \mathcal{N}(i)} \beta_l\,A_{i,j}^{l}\,\delta\big(\mathbf{h}'^{\,l,(t)}_j \mid \mathcal{V};\,j;\,\Delta\mathbf{d}_{i,j}\big)\,\mathbf{w}_{i,j}^{l} eq. 6
The paper says the depth features are aligned to the image stage size by stride max-pooling or interpolation §3.4. The released code does this with max-pooling for the deeper-to-shallower direction and nearest-neighbour upsampling by 2 for the other, and then a 1 × 1 convolution with batch norm and ReLU to 256 channels resnet_ddmp.py resnet_ddmp.py resnet_ddmp.py.
The pseudocode below is written from the paper and the released code together. Where they differ, the code wins and the difference is noted under Paper vs code.
# One DDMP block on one image stage (C = 256 after reduction) [§3.3] [§3.4]
# Inputs: stage_feats (B, Cs, H, W); depth_list: one (B, 256, H, W) per stage
# Output: refined stage_feats (B, Cs, H, W)
def ddmp_block(stage_feats, depth_list):
# Reduce image features to C = 256 [code: models/resnet_ddmp.py#L18-L20]
image_feats = reduce_1x1(stage_feats) # (B, 256, H, W)
messages = []
for stage, depth in enumerate(depth_list):
# Image walk: 18 offsets = 9 sample points x 2 coordinates [eq. 3]
# Image walk per stage [code: models/resnet_ddmp.py#L27-L30]
image_offsets = image_offset_conv[stage](image_feats) # (B, 18, H, W)
samples = deform_unfold(image_feats, image_offsets) # (B, C, 9, H, W)
# Depth walk: 18 offsets and 9 gate logits from one conv [§3.3]
# [code: models/resnet_ddmp.py#L173-L176]
depth_out = depth_conv[stage](depth) # (B, 27, H, W)
depth_offsets = depth_out[:, :18] # (B, 18, H, W)
depth_state = depth + deform_conv(depth, depth_offsets)
# Affinity over the 9 samples, per group, then softmax [eq. 5]
# [code: models/resnet_ddmp.py#L143-L148]
affinity = softmax_over_samples(relu(bn(conv3x3(depth_state))))
# One gate per sample from the depth logits, then sigmoid [eq. 5]
# [code: models/resnet_ddmp.py#L188-L190]
gate = sigmoid(depth_out[:, 18:]) # (B, 1, 9, H, W)
# Weighted sum over the 9 samples [eq. 4] [eq. 6]
message = sum_over_samples(samples * gate * affinity) # (B, C, H, W)
messages.append(message)
# Concatenate the image features and all stage messages [app. A]
fused = concat([image_feats] + messages, dim=1) # (B, 4C, H, W)
# Mix with a 1x1 conv, group norm and ReLU, then add the input [app. A]
# [code: models/resnet_ddmp.py#L47-L49] [code: models/resnet_ddmp.py#L160]
return stage_feats + relu(group_norm(conv1x1(fused)))Centre-aware depth feature encoding (CDE)
A head on the depth branch regresses the 3D centre of each object (the projected centre and the depth). It shares ground truth with the detection branch and guides the depth features to become instance-aware §3.5. The head is dropped at inference §3.5 fig. 2. The paper's objective adds this auxiliary term to the detection loss with a weight; the weight is studied in app. B and tab. 8.
Detection heads, anchors and objective
Anchors follow D4LCN §3.6. 2D anchors with IoU at least 0.5 against projected ground-truth boxes are positives §3.6. The predicted 2D and 3D boxes are parameterised as offsets from the anchor eq. 7:
[x,y]_{2d} = [\hat x,\hat y]_{2d} + [t_x,t_y]_{2d} \ast [\hat w,\hat h]_{2d} [w,h]_{2d} = [\hat w,\hat h]_{2d} \ast \exp([t_w,t_h]_{2d}) [x,y]_{p} = [\hat x,\hat y]_{p} + [t_x,t_y]_{p} \ast [\hat w,\hat h]_{2d} [w,h,l]_{3d} = [\hat w,\hat h,\hat l]_{3d} \ast \exp([t_w,t_h,t_l]_{3d}) [z,\theta]_{3d} = [\hat z,\hat\theta]_{3d} + [t_z,t_\theta]_{3d} eq. 7
The detection loss is classification plus 2D and 3D regression. Cross-entropy is used for classification, and smooth-L1 (\tilde{L}_1) for the boxes eq. 8 §3.6:
L_{det} = L_{cls} + L_{2d} + L_{3d} L_{2d} = \tilde{L}_1\big([t_x,t_y,t_w,t_h]_{2d},\,[t^{gt}_x,t^{gt}_y,t^{gt}_w,t^{gt}_h]_{2d}\big) L_{3d} = \tilde{L}_1\big([t_x,t_y]_{p},\,[t^{gt}_x,t^{gt}_y]_{p}\big) + \tilde{L}_1\big([t_z,t_w,t_h,t_l,t_\theta]_{3d},\,[t^{gt}_z,t^{gt}_w,t^{gt}_h,t^{gt}_l,t^{gt}_\theta]_{3d}\big) eq. 8
The depth-branch loss is eq. 9:
L_{dep} = \tilde{L}_1\big([t'_x,t'_y]_{p},\,[t^{gt}_x,t^{gt}_y]_{p}\big) + \tilde{L}_1\big([t'_z]_{3d},\,[t^{gt}_z]_{3d}\big) eq. 9
The total loss is a sum scaled by a focal-style factor on the classification score s_t, with \gamma = 0.5 eq. 10 §3.6:
L = (1 - s_t)^{\gamma}\,\big(L_{det} + L_{dep}\big), \quad \gamma = 0.5 eq. 10
# Training loss, as written in the paper [§3.6] [eq. 8-10]
# Inputs: detection outputs and depth-branch centre outputs for positive anchors
# Output: scalar loss
def training_loss(det_out, dep_out, targets, s_t):
# Detection loss: classification, 2D box, 3D box [eq. 8]
l_det = cls_loss(det_out, targets) + box_2d_loss(det_out, targets)
l_det = l_det + box_3d_loss(det_out, targets)
# Depth-branch centre loss, same targets as detection [eq. 9]
l_dep = centre_loss(dep_out, targets)
# Focal-style scale on the summed loss, gamma = 0.5 [eq. 10]
return (1 - s_t) ** 0.5 * (l_det + l_dep)Hyperparameters
Values in the paper are cited to their section. Values that appear only in the code are marked "code only" and cited to the file.
| Name | Value | Where | Notes |
|---|---|---|---|
| Dataset | KITTI, 7,481 train and 7,518 test images; val1 and val2 splits | §4 | Car is the reported class |
| Input size | 512 × 1760 | §4 | Config: test scale 512, crop 512 × 1760 depth_ddmp.py |
| Augmentation | horizontal flip, probability 0.5 | §4 | depth_ddmp.py |
| Backbone | ResNet-50, one for image, one for depth | §4 | depth_ddmp.py resnet_ddmp.py |
| Backbone pretraining | not stated | §4 | Code loads a COCO-trained Faster R-CNN ResNet-50 by default; see Pitfalls |
| Image normalisation | not stated | §4 | Code only: Caffe-style per-channel means 102.9801, 115.9465, 122.7717 (0–255 scale) and std 1 depth_ddmp.py |
| Depth normalisation | not stated | n/a | Code only: mean 4413.16, std 3270.02 for DORN depth depth_ddmp.py |
| Depth source | off-the-shelf monocular estimator (DORN) for most runs; PSMNet in one table | §4 tab. 6 | Released code uses the same DORN depth maps |
| Optimiser | SGD | §4 | core.py |
| Initial learning rate | 0.04 | §4 | depth_ddmp.py (0.01 × 4) |
| Momentum | 0.9 | §4 | depth_ddmp.py |
| Weight decay | 0.0005 | §4 | depth_ddmp.py |
| Learning-rate schedule | not stated | n/a | Code: one-cycle schedule with peak = initial rate and total steps = max_iter core.py; warm-up fraction 0.3, initial div 25, final div 1e4 lr.py |
| Batch size | 16 on 8 Tesla V100 | §4 | Config: 40 (5 per GPU); see Pitfalls depth_ddmp.py [README.md#L74] |
| Iterations | 36k | §4 | Config: 40,000 depth_ddmp.py |
| Dropout | not stated | n/a | Code only: 0.5 on early features; channel dropout 0.3 depth_ddmp.py resnet_ddmp.py |
| 2D anchor scales | 12 scales, 30 × 1.265^n for n = 0…11 | §4 | Code: 32 to 384 px; see Pitfalls depth_ddmp.py depth_ddmp.py |
| Aspect ratios | 0.5, 1.0, 1.5 (36 anchors total) | §4 | depth_ddmp.py |
| Positive anchors | 2D IoU at least 0.5 | §3.6 | Code: fg threshold 0.5 depth_ddmp.py |
| 3D anchors | initialised from mean statistics of matched 3D ground truth | §4 | Code: means and stds are None in config and computed at run time; not traced |
| NMS | 2D IoU threshold 0.4 | §4 | depth_ddmp.py |
| Pre-/post-NMS top-N | not stated | n/a | Code only: 3000 and 40 depth_ddmp.py |
| Feature channels C | 256 | §3.4 app. A | resnet_ddmp.py |
| Samples K | 3 × 3 = 9 | §3.3 app. A | Image and depth walks |
| Group size G | not stated (only "group size") | §3.4 | Code: 4 resnet_ddmp.py |
| Message steps T | 1 | §3.4 | Chosen to balance accuracy and speed |
| DDMP positions | Stage II and Stage III of the image trunk | §3.4 | resnet_ddmp.py resnet_ddmp.py |
| Loss weights (det : dep) | 1 : 1 gives the best moderate score on val | app. B tab. 8 | Released code weights both at 1 depth_ddmp.py |
| Focal exponent γ | 0.5 | eq. 10 | Code: exponent 1 on classification only; see Pitfalls depth_ddmp.py |
| Hard negatives | not stated | n/a | Code: on depth_ddmp.py |
| Sampling | not stated | n/a | Code: 20% foreground fraction, box samples 0.20 depth_ddmp.py |
| Classes | not stated for the Car tables | §4 | Code trains Car, Pedestrian and Cyclist, plus background depth_ddmp.py |
Architecture
The stage sizes come from the appendix and match the code's ResNet-50 layout. Image and depth trunks have the same shapes app. A tab. 7.
| Stage | Output (C × H × W) | Notes |
|---|---|---|
| conv1 | 64 × 256 × 880 | Stride 2 |
| conv2_x | 256 × 128 × 440 | Stride 2 |
| conv3_x (Stage II) | 512 × 64 × 220 | DDMP_1 is applied here |
| conv4_x (Stage III) | 1024 × 32 × 110 | DDMP_2 is applied here |
| conv5_x (Stage IV) | 2048 × 32 × 110 | Dilated conv5, stride 1 |
DDMP_1 takes image Stage II, depth Stages II, III and IV, and outputs 512 channels. The table gives the messages from each depth stage as 256 × 64 × 220, and the update as a 3 × 3 convolution over the concatenation app. A tab. 7. The released code uses a 1 × 1 convolution there (see Paper vs code).
Results
Author-reported throughout. All numbers are Car, IoU 0.7, unless marked. AP is in percent.
Test set tab. 1. AP3D and APBEV for Mod. / Easy / Hard, with R40 metrics. Speed and GPU are as reported by each row's source.
| Method | Venue | FPS | AP3D Mod | AP3D Easy | AP3D Hard | APBEV Mod | APBEV Easy | APBEV Hard | GPU |
|---|---|---|---|---|---|---|---|---|---|
| FQNet | CVPR 2019 | 2 | 1.51 | 2.77 | 1.01 | 3.23 | 5.40 | 2.46 | 1080Ti |
| ROI-10D | CVPR 2019 | 5 | 2.02 | 4.32 | 1.46 | 4.91 | 9.78 | 3.74 | - |
| MonoDIS | ICCV 2019 | - | 7.94 | 10.37 | 6.40 | 13.19 | 17.23 | 11.12 | Tesla V100 |
| MonoPair | CVPR 2020 | 17 | 9.99 | 13.04 | 8.65 | 14.83 | 19.28 | 12.89 | - |
| UR3D | ECCV 2020 | 8 | 8.61 | 15.58 | 6.00 | 12.51 | 21.85 | 9.20 | GTX Titan X |
| RTM3D | ECCV 2020 | 20 | 10.34 | 14.41 | 8.77 | 14.20 | 19.17 | 11.99 | 1080Ti |
| AM3D | ICCV 2019 | 3 | 10.74 | 16.50 | 9.52 | 17.32 | 25.30 | 14.91 | 1080Ti |
| DA-3Ddet | ECCV 2020 | 3 | 11.50 | 16.77 | 8.93 | 15.90 | 23.35 | 12.11 | Titan RTX |
| D4LCN | CVPR 2020 | 5 | 11.72 | 16.65 | 9.51 | 16.02 | 22.51 | 12.55 | 1080Ti |
| Kinematic3D | ECCV 2020 | 8 | 12.72 | 19.07 | 9.17 | 17.52 | 26.69 | 13.10 | - |
| This paper (DDMP-3D) | CVPR 2021 | 6 | 12.78 | 19.71 | 9.80 | 17.89 | 28.08 | 13.44 | Tesla V100 |
Validation splits tab. 2. Each cell is AP3D R40 / R11 at IoU 0.7. "-" means the source does not report it.
| Method | Val1 Mod | Val1 Easy | Val1 Hard | Val2 Mod | Val2 Easy | Val2 Hard |
|---|---|---|---|---|---|---|
| OFT-Net | - / 3.27 | - / 4.07 | - / 3.29 | - | - | - |
| FQNet | - / 5.50 | - / 5.98 | - / 4.75 | - / 5.11 | - / 5.45 | - / 4.45 |
| ROI-10D | - / 6.93 | - / 10.25 | - / 6.18 | - | - | - |
| MonoDIS | - / 7.60 | - / 11.06 | - / 6.37 | - | - | - |
| MonoGRNet | 7.56 / 10.19 | 11.90 / 13.88 | 5.76 / 7.62 | - | - | - |
| GS3D | - / 10.97 | - / 13.46 | - / 10.38 | - / 10.51 | - / 11.63 | - / 10.51 |
| shift R-CNN | - / 11.29 | - / 13.84 | - / 11.08 | - | - | - |
| MonoPSR | - / 11.48 | - / 12.75 | - / 8.59 | - / 12.24 | - / 13.94 | - / 10.77 |
| SS3D | - / 13.15 | - / 14.52 | - / 11.85 | - / 8.42 | - / 9.45 | - / 7.34 |
| RTM3D | - / 16.86 | - / 20.77 | - / 16.63 | - / 16.29 | - / 19.47 | - / 15.57 |
| M3D-RPN | 11.07 / 17.06 | 14.53 / 20.27 | 8.65 / 15.21 | 10.07 / 16.48 | 14.51 / 20.40 | 7.51 / 13.34 |
| Pseudo-LiDAR | - / 18.50 | - / 28.20 | - / 16.40 | - | - | - |
| Decoupled-3D | - / 18.68 | - / 26.95 | - / 15.82 | - | - | - |
| UR3D | 13.35 / 18.76 | 23.24 / 28.05 | 10.15 / 16.55 | 11.10 / 16.75 | 22.15 / 26.30 | 9.15 / 13.60 |
| AM3D | - / 21.09 | - / 32.23 | - / 17.26 | - | - | - |
| D4LCN | 16.20 / 21.71 | 22.32 / 26.97 | 12.30 / 18.22 | - / 19.54 | - / 24.29 | - / 16.38 |
| This paper (DDMP-3D) | 20.39 / 23.12 | 28.12 / 31.14 | 16.34 / 19.45 | 19.91 / 22.92 | 29.17 / 30.66 | 15.26 / 18.75 |
The text reports margins over D4LCN of 4.19 points on val (moderate) and 1.06 points on test (moderate) §4.1.
Cyclist and Pedestrian tab. 3. AP3D at IoU 0.5, val1 (R11) / test (R40), author-reported.
| Method | Cyclist Mod | Cyclist Easy | Cyclist Hard | Pedestrian Mod | Pedestrian Easy | Pedestrian Hard |
|---|---|---|---|---|---|---|
| D4LCN | 4.41 / 1.67 | 5.85 / 2.45 | 4.14 / 1.36 | 11.23 / 3.42 | 12.95 / 4.55 | 11.05 / 2.83 |
| Baseline | 4.07 / 0.17 | 4.50 / 0.32 | 4.08 / 0.17 | 6.93 / 2.32 | 8.11 / 3.05 | 6.78 / 1.81 |
| This paper | 6.47 / 2.50 | 8.01 / 4.18 | 6.27 / 2.32 | 12.11 / 3.55 | 14.42 / 4.93 | 12.05 / 3.01 |
Ablation on val1 tab. 4. AP3D and APBEV at IoU 0.7, Mod / Easy / Hard.
| Group | Setting | AP3D Mod | AP3D Easy | AP3D Hard | APBEV Mod | APBEV Easy | APBEV Hard |
|---|---|---|---|---|---|---|---|
| I | Baseline (dot product of image and depth features) | 18.82 | 26.03 | 16.27 | 24.18 | 33.06 | 19.63 |
| II | DDMP, single depth stage | 22.36 | 28.94 | 18.86 | 26.73 | 36.89 | 24.00 |
| III | DDMP, multi-scale depth | 22.84 | 28.12 | 19.09 | 27.05 | 37.11 | 24.20 |
| IV | DDMP, multi-scale, plus CDE | 23.12 | 31.14 | 19.45 | 27.46 | 37.71 | 24.53 |
Auxiliary depth task on val tab. 5. Which centre terms the depth head regresses.
| Depth-head target | AP3D Mod | AP3D Easy | AP3D Hard | APBEV Mod | APBEV Easy | APBEV Hard |
|---|---|---|---|---|---|---|
| None | 22.84 | 28.12 | 19.09 | 27.05 | 37.11 | 24.20 |
| z only | 23.07 | 28.11 | 19.19 | 26.95 | 36.70 | 24.06 |
| x and y | 22.51 | 28.68 | 18.52 | 26.80 | 37.15 | 21.38 |
| x, y and z (CDE) | 23.13 | 31.14 | 19.45 | 27.46 | 37.71 | 24.53 |
Depth estimator tab. 6. AP3D / APBEV, with DORN or PSMNet depth.
| Depth | Method | Mod | Easy | Hard |
|---|---|---|---|---|
| DORN | D4LCN | 21.71 / 19.54 | 26.97 / 24.29 | 18.22 / 16.38 |
| DORN | Baseline | 18.82 / 24.18 | 26.03 / 33.06 | 16.27 / 19.63 |
| DORN | This paper | 23.12 / 27.46 | 31.14 / 37.71 | 19.45 / 24.53 |
| PSMNet | D4LCN | 25.41 / - | 30.03 / - | 21.63 / - |
| PSMNet | Baseline | 25.41 / 33.34 | 35.26 / 46.75 | 20.69 / 27.27 |
| PSMNet | This paper | 30.83 / 36.20 | 41.76 / 52.87 | 24.78 / 29.34 |
Appendix results (val, AP3D Mod / Easy / Hard). Loss weights α : β between the detection and depth terms app. B tab. 8: 1:0 gives 22.84 / 28.12 / 19.09; 1:2 gives 22.71 / 31.35 / 18.94; 2:1 gives 22.85 / 32.32 / 19.35; 1:1 gives 23.13 / 31.14 / 19.45. Other depth-branch heads app. B tab. 9: bbox regression gives 22.48 / 28.84 / 18.31 and classification gives 22.69 / 28.72 / 19.16, against 22.84 / 28.12 / 19.09 with no extra head. Message variants app. B tab. 10: image-only 3DNet 14.61 / 17.94 / 12.74; 3DNet with DGMN 16.98 / 20.12 / 15.17; baseline with depth 18.82 / 26.03 / 16.27; 3DNet with DGMN and depth 19.59 / 27.78 / 16.48; DDMP 22.84 / 28.12 / 19.09; DDMP with CDE 23.13 / 31.14 / 19.45; DDMP with softmax affinities and CDE 23.17 / 32.40 / 19.35. The softmax row is the variant the released code matches.
Paper vs code
Repository: the paper's abstract links https://github.com/fudan-zvg/DDMP, which is taken as the official code. The repository's MIT licence names Li Wang, a first author, as copyright holder LICENSE. Commit 53ab4832917b674b9831a2c628cc63b2c29f983a. The README says the code is based on DGMN and thanks D4LCN [README.md#L23] [README.md#L102]. Pretrained weights are not in the repository; the README links one model on Google Drive [README.md#L97].
The configs train the 3-class detector; resnet_ddmp is the model for the depth config and resnet_depth_xyz for the xyz config depth_ddmp.py depth_ddmp_xyz.py.
Discrepancies:
- Affinity softmax. The main text does not normalise affinities. The appendix reports a softmax variant that is better, and the released code always applies softmax over the 9 samples resnet_ddmp.py tab. 10. The main-text headline (23.13) is the non-softmax row of tab. 10 app. B.
- Filter and affinity shapes. The appendix gives the weights as H × W × K × G app. A. The code uses K × G affinities (one per group) and a single K-vector gate shared across groups, multiplied with the sampled features resnet_ddmp.py resnet_ddmp.py resnet_ddmp.py.
- Gate. Equation 1 uses a transformation per node. The code's transformation is a per-group scalar weight, applied as a sigmoid gate, so there is no channel-wise matrix resnet_ddmp.py.
- Image walk. Equation 3 uses one walk for all depth stages. The code uses three image offset convolutions, one per depth stage resnet_ddmp.py resnet_ddmp.py.
- Stage weights. Equation 4 sums stage messages with \beta_l. The code has no \beta_l. It concatenates the image features and three stage messages, then applies a 1 × 1 convolution with group norm, ReLU and a residual resnet_ddmp.py resnet_ddmp.py. The appendix says a 3 × 3 convolution tab. 7.
- Depth gate. The code multiplies the final image-trunk feature by the depth Stage IV feature before the heads. This step is not described in the paper resnet_ddmp.py.
- Depth head. The paper says the centre head predicts without predefined anchors §3.5. The code's depth head predicts one value per anchor and trains on the same positive anchors as the detection head resnet_ddmp.py rpn_3d_multitask_center.py.
- Focal factor. Equation 10 uses γ = 0.5 on the whole sum. The config sets the focal exponent to 1 depth_ddmp.py. The code applies it as a per-sample weight on the classification loss only rpn_3d_multitask_center.py rpn_3d_multitask_center.py. A weight-sum is computed and never used rpn_3d_multitask_center.py.
- Classification clamp. The per-sample classification loss is clamped to [0, 2000] rpn_3d_multitask_center.py. The paper does not mention this.
- Depth-branch loss. The depth-branch loss is always added to the box loss, with weight 1 rpn_3d_multitask_center.py. Both the DDMP-only and DDMP-plus-CDE configs call the same loss with the depth head train.py. So the config alone cannot switch CDE off.
- Depth input for the CDE model. The xyz config feeds a 3-channel map: the depth value, the column index and the row index depth_ddmp_xyz.py imdb_util_xyz.py imdb_util_xyz.py. The paper describes a depth map. The README says this variant scores best, with 23.53 / 28.16 moderate [README.md#L97].
- Batch and iterations. The paper uses batch 16 on 8 GPUs and 36k iterations §4. The config uses batch 40 (5 per GPU) and 40,000 iterations depth_ddmp.py depth_ddmp.py [README.md#L74].
- Anchor scales. The paper uses 30 × 1.265^n §4. The code uses 32 to 384 pixels, a geometric series with ratio 12^(1/11) ≈ 1.2535 (computed by hand from the config) depth_ddmp.py depth_ddmp.py.
- Pretraining. The paper does not state it. The code loads a Faster R-CNN ResNet-50 trained on COCO, from
data/res50_faster_rcnn_iter_1190000.pthresnet_ddmp.py. The README says the backbone is ImageNet-pretrained [README.md#L76]. - Learning-rate schedule. Not stated in the paper. The code uses one-cycle core.py.
- Hill-climbing loss. The loss code has a hill-climbing term, but both configs leave it off depth_ddmp.py rpn_3d_multitask_center.py rpn_3d_multitask_center.py.
Pitfalls
- Depth-estimate dependency. Results depend on the depth map: DORN in most runs, PSMNet in tab. 6. The code's normalisation constants (mean 4413.16, std 3270.02) are the DORN values depth_ddmp.py. Swapping the depth model needs new constants; the paper does not give them.
- Depth trunk cost. The depth branch is a full second ResNet-50 resnet_ddmp.py. The paper's 6 FPS (Tesla V100, tab. 1) does not say whether the external depth model's time is counted.
- Ambiguous reported numbers. The paper gives 23.12 for val moderate AP3D R11 in tab. 2 and tab. 4, and 23.13 for what appears to be the same configuration in tab. 5 and tab. 8. This looks like a rounding or copy discrepancy in the paper. Check which one you are targeting.
- Softmax variant. As above, the main headline and the released code are different variants. Reproducing the main-text numbers means removing the softmax; reproducing the released code's numbers means keeping it tab. 10.
- Stage weights and fusion are unclear. β_l is not given a value, and the fusion layer is a 3 × 3 convolution in the appendix but a 1 × 1 in the code. Group size G is not stated in the paper; the code uses 4.
- s_t is undefined. The focal factor needs a per-sample classification score. The paper does not say whether it is per anchor, per object or averaged over the batch. The code uses per anchor.
- Focal exponent. The paper says 0.5; the code uses 1. A reimplementation following the paper will not match the code's loss scale.
- Depth-branch targets. The depth head shares the detection targets, so it is trained only on positives. The paper says it works without anchors.
- Ablation without CDE. The released code cannot reproduce group II or III (no CDE) by config, because the depth-branch loss is always on [scripts/train.py#L139]. You would have to edit the loss.
- Pretraining and normalisation. Not stated in the paper. Use the code's backbone weights and Caffe-style means if you want to match the released numbers, and say so.
- Class set. The paper does not say whether the Car tables come from a 3-class model or a Car-only model. The code trains 3 classes with a background class depth_ddmp.py resnet_ddmp.py.
- Runtime environment. The code needs maskrcnn_benchmark's deformable conv modules, a compiled NMS, CUDA (tensors are created on
torch.cudain the model) and a pinned old PyTorch (README: Python 3.8, PyTorch 1.4) resnet_ddmp.py [README.md#L19-L37]. Depth maps come from an external Google Drive link [README.md#L49]. - Hard negatives and sampling. The config enables hard negatives and a 20% foreground fraction depth_ddmp.py depth_ddmp.py. The paper is silent on sampling. I did not trace the sampler.
- Leaderboard claim. "Rank 1st" refers to the KITTI monocular track on the submission day, 16 November 2020 [abstract]. Later entries may differ. This is the authors' own claim and was not checked here.
Lineage
- Builds on: D4LCN (dynamic depth-dependent kernels from depth features) §1 §4.1; DGMN (dynamic graph message-passing networks, which the code is based on) §3.3 tab. 10 README.md; deformable convolution (learned sampling offsets) §3.3 app. A.
- Compared against: AM3D, pseudo-LiDAR methods, Kinematic3D, MonoPair, RTM3D, UR3D, DA-3Ddet and others tab. 1 tab. 2.
- Superseded by: not stated in this card.
Source and license
Adapted from “Depth-conditioned Dynamic Message Propagation for Monocular 3D Object Detection” by Li Wang, Liang Du, Xiaoqing Ye, Yanwei Fu, Guodong Guo, Xiangyang Xue, Jianfeng Feng, Li Zhang (arXiv:2103.16470), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/fudan-zvg/DDMP at commit 53ab4832917b (MIT). Any quoted code is shown under that license.