Monocular 3D Object Detection Leveraging Accurate Proposals and Shape Reconstruction

In plain words

This paper asks how far a single camera can go at finding objects in 3D, without a laser scanner. Its system starts from 2D boxes that an ordinary 2D detector draws around each object. It then uses the geometry of a camera: an object that looks small in the image must be far away, if its real size is roughly known. That gives a rough 3D position for each object, which the network then corrects.

The network also learns a small cloud of 3D points for each object, placed relative to the object's own centre. Laser data is used only during training, to build these targets. The training step checks that the cloud, once moved to its predicted place and projected back into the camera, lines up with the object's 2D box. This pushes the 3D position to agree with what the camera sees. At test time the system takes the camera image, its calibration and the 2D boxes.

TL;DR

MonoPSR is a monocular 3D detector for cars, pedestrians and cyclists. It builds one 3D centroid proposal per 2D detection from a pinhole depth relation, refines it by regression, and trains a per-object point cloud with LiDAR-derived targets. Its key result is state-of-the-art monocular car AP on the KITTI test server at the time, with 0.20 s runtime (author-reported). Use it as a reference for proposal-based monocular 3D detection, not as a drop-in model: several training values differ between the paper and the code.

Problem

Monocular 3D detection is under-constrained, because depth is lost when the scene is projected to the image §1. Earlier monocular methods either force the projected 3D box to fit the 2D box, so 2D errors pass into the 3D box §1, §2, or match objects to CAD models that are limited to the shapes they cover §1, §2. Methods that use depth only in training are rare §1. The paper reports that the best monocular car method then had AP about 67% lower than the best LiDAR method on the KITTI car class §1. Pedestrian and cyclist results were rarely reported for monocular methods §1.

Method

The network runs on the 2D detector's output boxes. For each box it builds a feature map from a crop of the object and from the whole image, shared by three heads: proposal generation, proposal refinement, and instance reconstruction §3. Inference uses only images, 2D boxes and calibration; LiDAR is used only to make training targets §3.3, §4.

Shared features §3.1. The instance crop and the whole image each pass through an encoder. Whole-image features are cropped per instance with RoI pooling and concatenated depth-wise with the crop features §3.1. Full-image features help the depth estimate §5.2.

Orientation and dimensions §3.1. The network predicts the observation angle β, the sum of the viewing angle α_h and the yaw θ, as discrete bins, following F-PointNet §3.1. The other two angles are fixed to zero for KITTI §3.1. Dimensions are offsets from class mean sizes §3.1.

Proposal depth §3.1 eq. 1. The depth proposal follows the pinhole relation: depth equals focal length times real height over the projected height in pixels eq. 1. The real height comes from the predicted dimensions, so this proposal is only as good as the height estimate §3.1.

Proposal location §3.1. The horizontal and vertical location of the proposal is the 2D box centre (u_c, v_c) re-projected at the proposal depth using calibration §3.1.

Refinement §3.2 eq. 2. The refinement stage regresses the depth as an offset s from the proposal depth. The offset depends on rotation, dimensions and viewing angles eq. 2. The paper argues that an offset from a good proposal is more stable than direct depth regression §3.2, §5.2. The lateral coordinate is not regressed: t_x is recovered from the ray through the estimated centroid, so it is not over-constrained §3.2. The regression targets are Δt_y and Δt_z §3.2.

Instance reconstruction §3.3 eq. 3. A small decoder predicts an L × W × 3 point cloud in an object coordinate frame, with the origin at the object centroid and rotated by α_h §3.3, fig. 3. Predicted clouds are masked to the visible part of the object §3.3. To train, LiDAR points inside each ground-truth box are interpolated, then transformed with a matrix T_CO built from α_h and the centroid eq. 3. That same LiDAR data gives instance masks, a depth map and the local cloud targets §3.3.

Losses §3.4 eq. 4. The total loss is the sum of centroid, orientation, dimension and reconstruction losses eq. 4. Regressions use smooth L1 and classifications use softmax §3.4. The paper states only that the weights make validation losses similar in size §3.4.

Reconstruction losses §3.4.1 fig. 4. Three losses act on the cloud. (1) Smooth L1 on the valid points of the local cloud. (2) A depth loss on the cloud's z-channel after transforming it by T_CO. (3) A projection alignment loss: the cloud is projected through the camera matrix and the error to the expected pixel grid, normalized by the 2D box width and height, is penalized with smooth L1 toward zero at valid pixels §3.4.1. This is the paper's main contribution: it links 3D shape and 2D appearance §1, §2.

Training §4. The 2D detector is MS-CNN; the full image is resized to 160 × 608 and each instance crop to 48 × 48 × 3 §4. The encoders start from ResNet-101 weights pretrained on KITTI 2D detection §4. Training uses Adam with an initial learning rate of 0.0008, decaying by 0.8 every 10K steps, for 100K iterations §4. Augmentation is only 2D box jitter: Gaussian noise on centre and size, scaled by box size, with at least 0.7 IoU to the original box §5.

Pseudocode of inference. The names follow the paper; P is the 3×4 camera matrix:

# MonoPSR inference for one image  [§3] [§4]
# Input:  image (3, H, W), 2D boxes (N, 4), camera matrix P (3, 4)
# Output: one 3D box per 2D box

def detect_3d(image, boxes_2d, P):
    # Crop each 2D box to 48 x 48; resize the full image to 160 x 608  [§4]
    crops = crop_and_resize(image, boxes_2d, (48, 48))
    full = resize(image, (160, 608))
    # Shared features: instance crop plus full-image RoI  [§3.1]
    feats = shared_features(crops, roi_crop(encode(full), boxes_2d))
    # Observation angle and dimension offsets  [§3.1]
    obs_angle, dims = orientation_and_dims_head(feats)
    # Viewing angle from the box centre ray (standard pinhole geometry;
    # not written out in the paper)  [§3.1]
    alpha_h = atan((boxes_centre_u - P[0, 2]) / P[0, 0])
    # Depth proposal from the pinhole relation  [§3.1] [eq. 1]
    prop_z = P[0, 0] * dims.height / box_height(boxes_2d)
    # Depth refinement: proposal plus learned offset  [§3.2] [eq. 2]
    cen_z = prop_z + depth_offset(feats, prop_z, dims, alpha_h)
    # Vertical location: box centre re-projected at cen_z, plus offset  [§3.1]
    prop_y = (boxes_centre_v - P[1, 2]) * cen_z / P[1, 1]
    cen_y = prop_y + offset_y(feats)
    # Horizontal location: ray at alpha_h through cen_z  [§3.2]
    # (official code: monopsr_output_builder.py#L559-L564)
    x_offset = -P[0, 3] / P[0, 0]
    cen_x = cen_z * tan(alpha_h) + x_offset
    return (cen_x, cen_y, cen_z), dims, obs_angle

The sketch omits the official code's inference-time cen_x post-processing, see Paper vs code.

The training loss, as code-configured weights, is:

# Training loss: four groups, weighted as in the official config  [§3.4] [eq. 4]
# Weights are from the code; the paper does not state them  [code: configs]
def total_loss(pred, gt):
    # Centroid offsets for y and z: smooth L1, weight 0.1 each
    loss = 0.1 * smooth_l1(pred.cen_y - gt.cen_y)
    loss += 0.1 * smooth_l1(pred.cen_z - gt.cen_z)
    # Orientation: bin softmax (weight 0.3) and smooth L1 residual (1.0)
    loss += orientation_loss(pred.alpha, gt.alpha)
    # Dimension offsets: smooth L1, weight 1.0
    loss += smooth_l1(pred.dims - gt.dims)
    # Reconstruction losses  [§3.4.1]
    loss += reconstruction_loss(pred, gt)
    return loss

# Instance reconstruction losses  [§3.4.1] [fig. 4]
# Weights from the config; the paper names the terms but not the weights
def reconstruction_loss(pred, gt):
    # Local point cloud, valid points only: smooth L1, weight 100.0
    loss = 100.0 * smooth_l1(pred.local_xyz[valid], gt.local_xyz[valid])
    # Depth map in the camera frame: smooth L1, weight 10.0
    loss += 10.0 * smooth_l1(pred.depth[valid], gt.depth[valid])
    # Projection alignment: weight 0.1
    loss += 0.1 * projection_loss(pred, gt)
    return loss

# Projection alignment loss  [§3.4.1] [code: monopsr_output_builder#L681]
# Error is normalized by the 2D box size; the code also clips it
def projection_loss(pred, gt):
    # Local points to camera frame with the rotation and centroid  [eq. 3]
    points_cam = transform(pred.local_xyz, alpha_h, pred.centroid)
    # Project with the camera matrix  [§3.4.1]
    uv_pred = project(P, points_cam)
    # Expected pixel grid of the 2D box (code: instance_utils)  [§3.4.1]
    error = (uv_expected - uv_pred) / (box_w, box_h)
    # Clip each normalized error to [-2, 2]; mask invalid pixels  [code]
    error = clip(error, -2.0, 2.0) * valid
    # Mean over valid pixels per instance, then smooth L1 toward 0  [§3.4.1]
    per_instance = sum(error) / max(count(valid), 1)
    return mean(smooth_l1(per_instance, 0.0))

Hyperparameters

Values from the paper are cited to the paper. Values from the official config are cited to the code. Where they differ, both are shown. See Paper vs code.

Results

Author-reported. KITTI 3D Object Detection benchmark. Two validation splits are used, val1 and val2, each following the split of a prior method; ablations use val1 §5. The 2D detector (MS-CNN) takes 80 ms and the network 120 ms on a Titan X §5.1. The validation tables report AP_BEV and AP_3D at 0.5 and 0.7 IoU, on the easy, moderate and hard levels §5.

Table 1. AP_BEV, KITTI car, val1 / val2 tab. 1

Mono3D and Deep3DBox numbers are taken from the MultiFusion paper §5.1.

Table 2. AP_3D, KITTI car, val1 / val2 tab. 2

Table 3. KITTI test server, car tab. 3 (the caption does not state the IoU)

Table 4. KITTI test server, pedestrians and cyclists tab. 4 The paper states that no other published method has results on this server for these classes.

Table 5. Pedestrians and cyclists, val1 tab. 5

Table 6. Proposal representation and full-image features tab. 6 Centroid depth error in metres (average absolute error / standard deviation), hard car, val1. Networks here have no reconstruction module.

C = RGB crop features; D = estimated 2D and 3D box dimensions; F = full-image features tab. 6.

Table 7. Instance reconstruction ablation tab. 7 AP_3D at 0.5 IoU, moderate car, val1.

Text claims the final row raises AP_3D at 0.5 IoU by 7.2% over the baseline §5.3. The table shows 34.47 to 41.71, an increase of 7.24 points.

Paper vs code

Official repository: kujason/monopsr, MIT license, at commit e194d2547f10e7ab50d786762dcf9f5619069ce8. The owner is the paper's first author, and the LICENSE names all three authors LICENSE. The README points to src/monopsr/configs/monopsr_model_000.yaml for training README.md. Each discrepancy below is checked against the code.

• Learning rate. Paper: 0.0008 §4. Code: 0.00008 monopsr_model_000.yaml.
• Training length. Paper: 100K iterations §4. Code: 142000 monopsr_model_000.yaml.
• Weight averaging. Not in the paper. Code keeps a moving average with decay 0.9999 monopsr_model_000.yaml.
• Projection frame. Paper: the cloud is placed using the estimated centroid and horizontal viewing angle §3.3. Code: the projection during training uses the ground-truth viewing angle, and the centroid from the predicted cen_y and cen_z monopsr_model.py (the projection branch's comments say "Calculate x from gt viewing angle").
• Vertical centroid. Paper: the proposal's vertical location is the 2D box centre re-projected at the proposal depth §3.1. Code: the same re-projection minus a fixed offset, which for Car is 0.0648 m under the KITTI trend setting instance_utils.py, applied in monopsr_output_builder.py.
• Horizontal centroid at inference. Not in the paper. Code re-computes cen_x from the predicted box after prediction, using the width ratio of the box and its projected corners instance_utils.py. The paper says t_x is recovered from the ray only §3.2.
• Projection normalization. Code clips the normalized error to [-2, 2] and averages over valid pixels monopsr_output_builder.py. Paper: normalized by box width and height, no clip or mean stated §3.4.1.
• Depth filter. Config: training objects are kept only for depth 5 to 45 m monopsr_model_000.yaml. Paper: KITTI depths range from 5 to 80 m §3.1. Paper does not give the training filter.
• Score reweighting. Code multiplies the detection score by a depth term, 1 − depth/45 clipped to [0.1, 1.0] monopsr_output_builder.py, through monopsr_model.py. Not in the paper.
• Mask. Paper calls the segmentation mask optional §3.3. Config trains it with weight 0.1 monopsr_model_000.yaml.
• Loss terms. Code uses 100.0, 10.0, 0.1 and others, plus loss terms that the paper's equation does not list. Paper gives no weights §3.4.
• Chamfer and EMD. The code has Chamfer and EMD losses, which need compiled CUDA ops losses_custom.py. The config uses them only as metrics, not as losses monopsr_model_000.yaml.
• Orientation residual. Code's alpha regression output is L2-normalized as a (sin, cos) pair and converted with atan2 monopsr_output_builder.py. Paper describes bin classification and bin-offset regression, as in F-PointNet §3.4, but not this parameterization.
• View angle loss. Config sets view_ang to "est", and the code adds a view-angle loss only for the "offset" option monopsr_model.py. So the viewing angle has no direct loss.
• Jitter. Code jitters only the oversampled boxes monopsr_model_000.yaml. Paper: "2D box jittering", with no parameters §5.
• Feature stride and full-image size. Paper says the crop feature map has stride 4 §4, the full image is "half of its original size" in §3.1 but 160 × 608 in §4, and the config sets the first-stage stride to 16 monopsr_model_000.yaml. Not resolved from code.

Pitfalls

• Learning rate and steps. The paper's 0.0008 and 100K steps do not match the code's 0.00008 and 142000 steps. An implementation that follows only the paper will train at a rate 10 times higher than the official config.
• Projection with GT angle. The official code projects the cloud with the ground-truth viewing angle during training monopsr_model.py, while the paper describes the predicted one §3.3. The two differ whenever the predicted angle is wrong, so the projection loss in training does not see the angle the network will use at test time.
• Test-time x. Inference applies a cen_x post-processing step that the paper does not describe monopsr_model_000.yaml. The paper's numbers do not say whether it was used.
• Depth scoring. Scores are reweighted by depth in the code, which can change ranking and AP. Not in the paper.
• Unstated values. Number of orientation bins, FC sizes, box jitter standard deviations, loss weights, the training difficulty filter, the moving average and the batch size are not stated in the paper. Values listed in this card come from the config.
• Crop and image sizes. The paper contradicts itself on the full-image size (§3.1 vs §4). The code uses 160 × 608 monopsr_model_000.yaml.
• Stride. The 12 × 12 feature map is consistent with stride 4 §4, while the config sets 16. An implementer should check the feature extractor before choosing.
• Percentage points. The "7.2%" gain in §5.3 is 7.24 points of AP_3D (34.47 to 41.71), not a relative 21%.
• BEV not best everywhere. On val1 at 0.7 IoU, easy level, MultiFusion has higher BEV AP (22.03 vs 20.63) tab. 1.
• Runtime. Table 3 lists 0.20 s. The text gives 120 ms for the network plus 80 ms for the 2D detector §5.1.
• Pedestrians and cyclists. These results are only on the test server with no other published comparison tab. 4. The paper's claim of first results is the author's §5.1. Figures show that pedestrian 3D boxes do not fit their 2D boxes, which is why a tight-fit prior is avoided fig. 5.
• Dependencies. The code uses TensorFlow 1.x APIs such as tf.to_float losses_custom.py, needs compiled custom CUDA ops, and the README sets up a Python 3.5 environment README.md. Training data also needs interpolated LiDAR and instance masks made with separate demo scripts . Plan these as a separate pipeline.
• Data requirements. Training needs LiDAR and the derived depth and mask maps. Inference needs only the 2D detector output and calibration §3.3, §4.

Lineage

• Builds on: Deep3DBox for the observation-angle parameterisation, whose tight-fit 2D constraint the paper argues against §3.1, §2; F-PointNet for the discrete-bin orientation loss and local frames §3.1, §3.4, §3.3; Faster R-CNN for two-stage proposal regression §1; MS-CNN as the 2D detector §4; MultiFusion for using LiDAR-derived depth in training §1.
• Compared against: Mono3D, Deep3DBox, A3DODWTDA and MultiFusion on KITTI val and test tab. 1 to 3.
• Superseded by: not stated in this wiki.