Proposal refinement

Overview

Proposal refinement is a two-step estimate. A first step produces a rough value from a rule or a simple head. A second, learned step predicts a correction to that value. In monocular 3D detection the rough value is a depth from the pinhole camera relation, and the learned step adjusts it 1904.01690 §3.1 1904.01690 §3.2. In LiDAR 3D detection the rough value is a whole 3D box from any detector, and the learned step corrects its centre, size and heading 2103.15297 §1 2103.15297 §3.1.

The invariant is that the learned step predicts a change relative to the proposal, not the absolute value. The monocular paper argues that an offset from a good proposal is more stable to learn than a direct estimate 1904.01690 §3.2 1904.01690 §5.2. The LiDAR paper adds a condition: the refiner must be able to see the proposal's size, or it cannot correct it 2103.15297 §3.2.

Variants

Pinhole-prior depth proposal with a learned offset

The depth proposal is the focal length times the real object height, divided by the height in pixels of the 2D box 1904.01690 eq. 1. The real height comes from a predicted size, so the proposal is only as good as that size estimate 1904.01690 §3.1. The refinement stage predicts an offset from the proposal depth. The offset depends on rotation, dimensions and viewing angles 1904.01690 eq. 2. The lateral position is not regressed. It is read off the viewing ray through the estimated centre 1904.01690 §3.2. Card: Monocular 3D Object Detection Leveraging Accurate Proposals and Shape Reconstruction.

The paper's depth-error ablation, on hard cars in val1 with no reconstruction module, gives 1.22 m for proposal regression with full-image features, against 1.48 m for direct estimation with the same inputs 1904.01690 tab. 6. These are single-split, author-reported numbers.

LiDAR box refinement with a boundary offset

The second stage takes a 3D proposal (x, y, z, w, l, h, θ) from a LiDAR detector and reads the raw points inside an enlarged box around it. The points are moved into the box frame, and each point gets six extra values: its distance to the two faces on each axis 2103.15297 §3.1 2103.15297 §3.3. A small PointNet predicts a class score and corrected centre, size and heading. The centre offset is divided by the proposal's w, l and h; the size target is a log ratio; the heading target is the smaller of two residuals, so a 180° flip is not penalised 2103.15297 eq. 4 2103.15297 eq. 5 2103.15297 eq. 7. Card: LiDAR R-CNN: An Efficient and Universal 3D Object Detector.

On the Waymo validation set, the paper reports vehicle 3D AP of 75.6 with PointPillars proposals, against 72.1 for the PointPillars baseline 2103.15297 tab. 1. Its size ablation puts the boundary-offset input at 75.6, against 74.1 for plain point refinement without the offset 2103.15297 tab. 4. These are author-reported numbers on one dataset.

The paper also measures the problem it targets. After enlarging PointPillars boxes by 1 m in length and width, 21.3% of the test boxes kept the same point count 2103.15297 §4.2. The base detector is not re-trained 2103.15297 §1.

When to use which

Neither paper compares region-proposal or anchor-based proposals as alternatives to a learned refinement. Add cards before using this page to choose between them.

Implementation sketch

The first sketch follows the monocular depth path only. The paper's full pipeline also predicts the vertical and lateral position and the 3D shape 1904.01690 §3.1 1904.01690 §3.3.

# Depth proposal from the pinhole relation  [arxiv:1904.01690 eq. 1]
# Inputs: focal length f in pixels, predicted real height H in metres,
# projected 2D box height h in pixels
def propose_depth(f, H, h):
    # Depth is focal length times real height over projected height
    return f * H / h

# Learned correction to the proposal  [arxiv:1904.01690 §3.2] [arxiv:1904.01690 eq. 2]
def refine_depth(z_proposal, features):
    # Offset depends on rotation, dimensions and viewing angles
    s = offset_head(features)
    return z_proposal + s

The second sketch covers the LiDAR box path. The points are used in the box frame, and the targets are normalised by the proposal's size 2103.15297 §3.1 2103.15297 §3.3.

# Refine one 3D proposal  [arxiv:2103.15297 §3.1] [arxiv:2103.15297 §3.3]
# Input:  points (N, 3); proposal (x, y, z, w, l, h, θ)
# Output: class score, corrected centre, size and heading

def refine_box(points, proposal):
    # Keep points inside the proposal's width and length, enlarged  [§3.1]
    local = to_box_frame(points_in_enlarged_bev(points, proposal), proposal)
    # Boundary offset: distance to the two faces on each axis  [§3.3]
    half = proposal.size / 2
    features = concatenate(local, half - local, half + local)
    # Fixed number of points per proposal  [§4.3]
    pooled = max_over_points(shared_mlp(sample_points(features)))
    hidden = fully_connected(pooled)
    return class_head(hidden), centre_head(hidden), size_head(hidden), heading_head(hidden)

Contradictions

• Lateral position (monocular). The paper says the lateral coordinate is recovered from the viewing ray and not regressed 1904.01690 §3.2. The official code recomputes it at inference from the predicted box, a step the paper does not describe (see the monocular card's Paper vs code section). Which one the reported numbers use is not resolved.
• Vertical position (monocular). The paper places the proposal at the re-projected box centre 1904.01690 §3.1. The official code subtracts a fixed offset for cars, which the paper does not describe (see the monocular card's Paper vs code section).
• Offset versus direct regression. The monocular paper argues an offset from a proposal is more stable than direct regression 1904.01690 §3.2 1904.01690 §5.2. The LiDAR paper does not test offset against direct regression. Its ablations vary the input encoding and which targets are refined 2103.15297 tab. 4 2103.15297 tab. 8. The two results measure different things, depth error from images and AP from point clouds, so they cannot be ranked against each other.

Papers

• Monocular 3D Object Detection Leveraging Accurate Proposals and Shape Reconstruction: a pinhole depth proposal, refined by a learned offset, with a per-object point cloud trained from LiDAR-derived targets.
• LiDAR R-CNN: An Efficient and Universal 3D Object Detector: a PointNet second stage that refines boxes from any LiDAR detector, with boundary-offset features that let it see proposal size.