Disentangling Monocular 3D Object Detection

In plain words

From one photo, a camera cannot tell how far away a car is directly, so 3D car detection from a single image is hard. A common approach predicts several kinds of numbers for each car (its distance, its size, where it sits in the image, and which way it faces) and adds all their errors into one training loss. These numbers have different units, so their errors pull against each other and training can become unstable.

The paper's main idea is to split each loss into several copies, one per group of numbers. In each copy only that group comes from the prediction, and the rest is set to the correct answer. Each copy is still the same kind of loss, so the groups are on comparable terms, and the whole network can train in one go without a staged schedule. The paper also adds a confidence score for each 3D box that the network learns without extra labels, and a 2D loss based on an overlap measure that stays useful when boxes do not touch.

The paper also points out a flaw in the average-precision formula used by the KITTI benchmark: a single correct guess per difficulty level already scores about 9%. The authors change the formula to use 40 recall points instead of 11, and they report results on KITTI and nuScenes with that change. For an engineer, it is a useful read on how car-detection numbers are computed.

TL;DR

A two-stage monocular 3D car detector (ResNet34 + FPN, RetinaNet-style 2D head, 3D head with 10 outputs) trained end-to-end. The key change is a disentangling transformation that turns each loss into one term per parameter group, which the authors report trains faster and more stably. It also adds a 3D confidence trained from the 3D box error and a signed IoU loss for 2D boxes. The paper shows that the KITTI 11-point AP gives a single correct detection about 9% AP, and it reports results using a 40-point AP instead. Author-reported results are state of the art on the KITTI car class. No official code was found.

Problem

Monocular 3D detection is ill-posed: without depth, 3D-to-2D mappings are ambiguous, and results from a single RGB image lag behind LiDAR- or stereo-based methods §1. The parameters of a 3D box (depth, size, rotation, projected centre) have different units and meanings. When their errors are added in one loss, the optimisation dynamics suffer, and prior monocular methods train in stages, first the 2D detector and then the 3D reasoning §1. The method needs per-image calibration at training and test time, and it uses only RGB images and 3D box labels §3. Separately, the 11-point interpolated AP used on KITTI gives a single correct detection per difficulty level about 9% AP on the dataset level, up to three times the performance of some recent works §1 §6.

Method

Overview. A single-stage 2D detector (RetinaNet-style head) runs on a ResNet34 + FPN backbone. A second-stage 3D head regresses one 3D box for each surviving 2D box §4. The 2D box loss and the 3D box loss are disentangled, and the 3D head also predicts a confidence for each 3D box §5 §4.3.

1. Backbone §4.1. ResNet34 with a feature pyramid. The BatchNorm + ReLU layers are replaced by a synchronized InPlaceABN activated with LeakyReLU of negative slope 0.01 §4.1. The authors state this does not change accuracy and frees GPU memory for larger batches or inputs §4.1. The FPN is connected to ResNet stages conv3, conv4 and conv5 (downsampling ×8, ×16, ×32) and gives five output maps (×8 to ×128), each with 256 channels, using 3×3 convolutions followed by synchronized InPlaceABN §4.1 fig. 2.

2. 2D head §4.2. The same head is applied to each of the five maps, and its parameters are shared across scales. It has two parallel stacks of 3×3 convolutions and produces, per cell and per anchor, a confidence and a 4-value box delta §4.2 fig. 3. For a cell at image position (u_g, v_g) and an anchor of size (w_a, h_a), the decoded box is:

• confidence p_2D = sigmoid(ζ_2D) §4.2;
• centre (u_b, v_b) = (u_g + δ_u · w_a, v_g + δ_v · h_a) §4.2;
• size (w_b, h_b) = (w_a · e^{δ_w}, h_a · e^{δ_h}) §4.2.

The confidence is trained with focal loss, which has weighting parameters α and γ whose values are not stated §4.2. The confidence target is 1 when the anchor's reference box has an IoU above τ_iou with a ground-truth box, and 0 otherwise §4.2 §7.2. Each positive anchor's box is trained with the signed IoU loss of eq. 1 §4.2. Output filtering follows RetinaNet: scores below 0.05 are dropped, then non-maximum suppression with IoU 0.5 runs on the top 5000 scores and the best 100 are kept §4.2.

3. 3D head §4.3. For each surviving 2D box, ROIAlign pools features into a 14×14 grid, then 2×2 average pooling gives a 7×7×128 map §4.3. The FPN level is chosen by the box size as k = min(5, max(1, floor(2 + log2(sqrt(w_b·h_b)/224)))) §4.3. Two parallel branches of fully connected layers, each 512 wide, read the map. All but the last layer of each branch are followed by non-synchronized InPlaceABN §4.3 fig. 4. The head returns a 10-value vector θ = (δz, Δu, Δv, δW, δH, δD, q_r, q_i, q_j, q_k) and a logit ζ_3D §4.3:

• depth z = μ_z + σ_z · δz, with dataset-wide depth statistics μ_z and σ_z §4.3;
• projected centre c = (u_b + Δu, v_b + Δv) §4.3;
• size s = (W_0 · e^{δW}, H_0 · e^{δH}, D_0 · e^{δD}), with dataset-wide reference size (W_0, H_0, D_0) §4.3;
• an allocentric quaternion q = q_r + q_i·i + q_j·j + q_k·k §4.3;
• 3D confidence p_3D|2D = sigmoid(ζ_3D), conditioned on a valid 2D box §4.3.

The 10 values become a 3D box with the lifting transform F app. B. That transform builds the centre C from z and c with the intrinsics K, multiplies the allocentric quaternion by a rotation about the vertical axis by β = atan(C_x / C_z), and places the unit cube scaled by s at C app. B:

\mathcal{F}(\theta) = \tfrac{1}{2} R_{q_c} S B_0 + C app. B

with q_c = q [cos(β/2) + sin(β/2) j], S = diag(s) and B_0 the eight corners of [-1, 1]^3 app. B. The 3D box loss compares the predicted corners B with the ground-truth corners B̂ (3×8):

L^{bb}_{3D}(B, \hat B) = \tfrac{1}{8} \lVert B - \hat B \rVert_H eq. 2

where ‖·‖_H is the Huber loss applied to each element, with δ_H = 3.0 §4.3 §7.2.

4. Self-supervised 3D confidence §4.3. The target is the 3D box loss turned into a probability, p̂_3D|2D = exp(−L^bb_3D / T) with temperature T = 1 §4.3 §7.2. The confidence is trained with binary cross-entropy against that target:

L^{conf}_{3D}(p, \hat p) = -\hat p \log p - (1 - \hat p)\log(1 - p) §4.3

The text says the confidence "will be steered by" the entangled 3D box loss of eq. 2, not by its disentangled variant §4.3.

5. Final score §4.3. The final 3D confidence is the product of the 3D conditional and the 2D confidence, p_3D = p_3D|2D · p_2D. Detections are kept when p_3D reaches τ_conf = 0.05 §4.3 §7.2. No NMS is applied to the 3D boxes, and no 3D priors are used to filter them (for example, flying cars are not removed) §4.3.

6. Disentangling transformation §5.1. Let L be a box loss with the property L(y, ŷ) = 0 when ŷ = y, and let ψ be a one-to-one map from the network output θ to the box y (for 2D, the 4D centre-plus-size parametrization; for 3D, the lifting F) §5.1. Split the output dimensions into k groups, let θ_j be the j-th group and θ_{−j} the rest. The disentangled loss is:

L_{dis}(y, \hat y) = \sum_{j=1}^{k} L\big(\psi(\theta_j, \hat\theta_{-j}), \hat y\big) §5.1

where θ = ψ⁻¹(y) and θ̂ = ψ⁻¹(ŷ). Each term keeps group j from the prediction and fixes all other groups to the ground-truth parametrization §5.1. The groups used are: for 2D boxes, the size and the centre (k = 2); for 3D boxes, depth, projected centre, rotation and dimensions (k = 4) §5.1. The transformation is applied to both the 2D loss eq. 1 and the 3D loss eq. 2 in all main experiments, unless stated otherwise §5.1.

7. 2D signed IoU loss eq. 1 app. A. The 2D box loss is L^bb_2D = 1 − sIoU(b, b̂) eq. 1. The signed IoU extends IoU so that disjoint boxes get a negative intersection and so keep a gradient §4.2 app. A:

\mathrm{sIoU}(b, \hat b) = \frac{|b \sqcap \hat b|_{\pm}}{|b| + |\hat b| - |b \sqcap \hat b|_{\pm}} eq. 3

The intersection box takes the max of the top-left corners and the min of the bottom-right corners. Its signed area is positive when the second corner is bottom-right of the first, and negative otherwise app. A. The score lies in [−1, 1] app. A.

8. Training schedule §7.2. The loss weights are 1.0 for every 2D head loss and 0.5 for every 3D head loss §7.2. Optimiser: SGD with learning rate 0.01, weight decay 0.0001 on all parameters except the scale and bias of InPlaceABN, conv1 and conv2 of ResNet34 frozen, batch size 96 on 4 V100 GPUs, 20k iterations, learning rate ×0.1 at 12k and 16k iterations §7.2. Augmentation: horizontal flip only §7.2.

# MonoDIS training loss for one image  [§4, §5, §7.2]
# Input:  image (3, H, W); calibration K (3, 3); ground-truth 2D and 3D boxes
# Output: scalar loss

def training_loss(image, K, gt_2d, gt_3d):
    # 1. Backbone: ResNet34 with FPN gives five scales  [§4.1]
    feats = fpn(resnet34(image))              # f_1..f_5, 256 channels each
    # 2. 2D head: one confidence and one 4-value box per anchor  [§4.2]
    conf_2d, box_2d = head_2d(feats)
    # 3. Anchors with IoU > 0.5 to a ground-truth box are positive  [§7.2]
    pos, neg = assign_anchors(gt_2d, iou_threshold=0.5)
    # 4. Focal loss on the 2D confidence  [§4.2]
    loss_conf_2d = focal_loss(conf_2d, pos, neg)
    # 5. Signed-IoU box loss, disentangled over size and centre  [eq. 1] [§5.1]
    loss_box_2d = 0
    for group in ["size", "centre"]:
        # Only this group comes from the prediction; the rest is ground truth
        mixed = take_group(box_2d[pos], gt_2d[pos], group)
        loss_box_2d += 1 - siou(mixed, gt_2d[pos])
    # 6. Positive 2D boxes: ROIAlign on the FPN level for their size  [§4.3]
    #    (which boxes train the 3D head is not stated)
    pooled = avg_pool_2x2(roi_align(feats, boxes_2d[pos], out_size=14))
    # 7. 3D head: 10 box values and one 3D confidence logit  [§4.3]
    params, zeta_3d = head_3d(pooled)
    gt_params = inverse_lift(gt_3d, K)        # psi^-1 of the ground truth
    # 8. Disentangled 3D box loss over four parameter groups  [§5.1] [eq. 2]
    loss_box_3d = 0
    for group in ["depth", "centre", "rotation", "size"]:
        mixed = take_group(params, gt_params, group)
        boxes = lift(mixed, K)                # lifting transform  [app. B]
        loss_box_3d += huber_corners(boxes, gt_3d, delta=3.0) / 8
    # 9. Self-supervised 3D confidence from the entangled box loss  [§4.3]
    entangled = huber_corners(lift(params, K), gt_3d, delta=3.0) / 8
    target = exp(-entangled / 1.0)            # temperature T = 1  [§7.2]
    loss_conf_3d = bce(sigmoid(zeta_3d), target)
    # 10. Weights: 1.0 for the 2D head, 0.5 for the 3D head  [§7.2]
    return (loss_conf_2d + loss_box_2d
            + 0.5 * (loss_box_3d + loss_conf_3d))

Hyperparameters

Values for the main KITTI car model unless the row says otherwise. "not stated" means the paper does not give it, and no official code was found to check.

Results

Author-reported. Setup: KITTI3D car class, IoU 0.7, unless noted. AP11 is the original 11-point metric and AP40 is the authors' 40-point variant §6. Each table is listed with its number from the LaTeX source order; they were not checked against a compiled PDF.

Headline (KITTI test, car, 3D AP40, Easy / Moderate / Hard) tab. 3: MonoDIS, larger training split: 10.37 / 7.94 / 6.40. MonoDIS, standard split: 7.03 / 4.89 / 4.08. MonoGRNet (described in §2 as the state of the art for RGB-only input): 9.61 / 5.74 / 4.25. The "larger training split" is defined only by a footnote link to a third-party repository §7.4.

Headline (KITTI validation, car, 3D AP11, Easy / Moderate / Hard) tab. 5: MonoDIS 18.05 / 14.98 / 13.42. MonoGRNet 13.88 / 10.19 / 7.62. Single correct hypothesis per difficulty 9.09 / 9.09 / 9.09 tab. 5.

Table 1. 2D detection, AP40 (2D ablation) §7.3. Evaluation split not stated.

Table 2. Confidence choice, validation split of Chen et al. §7.4. The AP40 block's "random" row is labelled AP11 in the source (see Pitfalls).

Table 3. AP40, validation ablations (white rows) and test-set SOTA (grey) and MonoDIS (green) tab. 3. Validation split of Chen et al. for ablation rows. Test rows follow the KITTI test server; the authors computed AP40 for test from the published precision-recall values §7.4.

Table 4. AP11, KITTI test set tab. 4. Grey: SOTA. Green: MonoDIS.

Table 5. AP11, KITTI validation set, ablations (white) and SOTA (grey) tab. 5.

Table 6. Pedestrian and cyclist, KITTI validation, IoU 0.5 tab. 6. Trained independently; the authors attribute lower scores to about 6× fewer pedestrian and 20× fewer cyclist annotations than cars, and to larger localisation impact on small objects §7.4.

Table 7. nuScenes, car class, 2 m matching threshold unless noted tab. 7. Author-reported. The paper reports AP (with matches on 2D ground-plane centre distance) and TP errors. It does not report NDS, because that needs attribute and velocity outputs the model lacks §8.

Headline summary the authors draw §8: about 42 points absolute AP at 2 m over OFTNet, better than OFTNet on all three true-positive errors, and better than PointPillar on two of three. The authors caution that only OFTNet is a fair comparison, because PointPillar uses LiDAR.

Confidence and metric effects §6 §7.4. A single correct prediction per difficulty level gives AP11 = 1/11 ≈ 0.0909 (9.09%) §6 tab. 5. Replacing p_3D with p_2D, or with random confidence, cuts AP40 moderate from 7.60 to 3.72 and 1.51 tab. 2.

Paper vs code

No official code.

The paper gives no code link. The project page (research.mapillary.com/publication/MonoDIS) links only the Mapillary GitHub organisation, not a repository for this paper §7.4 [fn. in §7.4]. Searches of the mapillary organisation for "MonoDIS", "kitti" and "monocular 3d detection" returned nothing, and a repository-wide search for "MonoDIS" returned only third-party copies, which are not used here.

Pitfalls

• Metric flaw and the fix. The 11-point AP takes precision at recall 0.0, so one correct detection at the lowest recall bin gives 1/11 per difficulty level §6. The authors use 40 recall points, {1/40, …, 1}, which drops recall 0 §6. To reproduce AP40, use the precision-recall points from the official evaluation file: the paper computed its test AP40 values from the leaderboard PR values §7.4 fn..
• "With a single exception" is not named. The text claims the approach beats all baselines on all 3D and BEV metrics except one §7.4. Table 3 shows the standard-split test row trailing MonoGRNet on every 3D and BEV column, and the larger-split row trailing it on BEV Easy (17.23 vs 18.19) tab. 3. Treat the "by large margins" claim with care.
• Loss reduction is not fully specified. Eq. 2 has ‖·‖_H applied to each element of the 3×8 matrix, and a 1/8 factor. The paper does not say whether the elements are summed or averaged before the 1/8 §4.3. The Huber δ_H = 3.0 has no unit in the text, and the reference sizes are in metres §7.2.
• Disentangled loss magnitude. L_dis is a sum of k terms, each a full-size box loss §5.1. The 3D version therefore has four terms and the 2D version two. Whether the sum is normalised is not stated.
• Confidence target. The 3D confidence target comes from the entangled loss eq. 2, not the disentangled one §4.3. The paper does not say whether the target is detached from the gradient. The self-supervised label is described only as "steered" by eq. 2 §4.3.
• Training boxes for the 3D head. The text does not say which 2D proposals receive 3D targets during training, or how a proposal is matched to a ground-truth box §4.3. Testing uses the filtered 2D boxes §4.3. The pseudocode above marks this as a choice that is not stated.
• Learning-rate schedule. §7.2 says the schedule "does not involve any multi-step or warm-up procedures" and also sets ×0.1 steps at 12k and 16k iterations §7.2. Implement the steps as written, and do not add a warm-up.
• Frozen layers. "conv1 and conv2 of ResNet34" are frozen §7.2. The text does not say whether batch-norm statistics are frozen with them, or what "conv2" maps to in the ResNet34 layout.
• Channel count in the 3D head. FPN outputs have 256 channels §4.1, ROIAlign output is 7×7×128 §4.3, and the text does not describe how 256 becomes 128. fig. 4 may show it; this was not checked.
• Focal loss constants. α and γ are never given §4.2. They are needed to reproduce the 2D confidence.
• Input resolution. The resolution is "set according to" the ROI-10D paper §7.2, and the value is not in this card. Results depend on it.
• Post-processing. 2D boxes get NMS; 3D boxes do not §4.2 §4.3. Duplicate 3D boxes can survive if their 2D boxes differ slightly. The paper does not describe how the final list is built beyond the τ_conf threshold.
• Calibration is required. The lifting needs the intrinsics K at training and test time §3 app. B. Images without calibration cannot be used.
• Table 2 labelling. The "random" row in the AP40 block says "AP11" in the source tab. 2. Read it as AP40 with caution.
• Table 1 split. The evaluation split for the 2D ablation is not stated §7.3.
• Larger training split. Its membership is only given by a footnote link to a third-party repository §7.4. Reproduce the standard split first.
• Pedestrian and cyclist. Their training settings are not given beyond "trained independently" §7.4.
• nuScenes. No training schedule, input size or validation split is given for nuScenes §8. Only the metric definitions are stated.
• Cost. 4 V100 GPUs, batch 96, 20k iterations §7.2. InPlaceABN saves GPU memory and the authors use it to allow larger batches or inputs §4.1.
• Toy experiment. The advantage of the disentangled loss on the toy problem is shown in trajectories and not in a table §5.2 fig. 7.

Lineage

• Builds on: ROI-10D (Manhardt et al., CVPR 2019) for the 10-value 3D parametrisation, the lifting transform and the allocentric pose §4.3 app. B. Also RetinaNet (focal loss and the 2D head, [Lin+17]) §4.2, FPN ([Lin2016]) §4.1, and synchronized InPlaceABN ([RotPorKon18a]) §4.1.
• Compared against: OFTNet ([Roddick18]), FQNet ([Liu+19]), MonoGRNet ([qin2019monogrnet]), Mono3D ([Chen_2016_CVPR]), Mono3D++ ([TongHe_2019_arxiv]), Xu et al. ([Xu_2018_CVPR]), Barabanau et al. ([Barabanau_arXiv_2019]), with PointPillar ([Lang_CVPR_2019]) on nuScenes tab. 3 tab. 5 tab. 7.
• Superseded by: none in this wiki.