---
kind: concept
slug: proposal-refinement
title: Proposal refinement
summary: >-
  A cheap first estimate is corrected by a learned second stage. Two cards: a monocular depth proposal from a pinhole relation with a learned offset, and a LiDAR second stage that refines whole 3D boxes and adds each point's distance to the box faces. Author-reported gains: 1.22 m vs 1.48 m depth error; 75.6 vs 72.1 vehicle AP on Waymo validation.
tags: [object-detection, 3d-object-detection, monocular-vision, lidar, point-cloud, regression]
papers: ["1904.01690", "2103.15297"]
related: [object-detection, bounding-box-regression-loss, monocular-depth-estimation, multi-task-learning]
status: draft
updated_at: 2026-11-06
---

## Overview

Proposal refinement is a two-step estimate. A first step produces a rough
value from a rule or a simple head. A second, learned step predicts a
correction to that value. In monocular 3D detection the rough value is a depth
from the pinhole camera relation, and the learned step adjusts it
[arxiv:1904.01690 §3.1] [arxiv:1904.01690 §3.2]. In LiDAR 3D detection the rough
value is a whole 3D box from any detector, and the learned step corrects its
centre, size and heading [arxiv:2103.15297 §1] [arxiv:2103.15297 §3.1].

The invariant is that the learned step predicts a change relative to the
proposal, not the absolute value. The monocular paper argues that an offset
from a good proposal is more stable to learn than a direct estimate
[arxiv:1904.01690 §3.2] [arxiv:1904.01690 §5.2]. The LiDAR paper adds a
condition: the refiner must be able to see the proposal's size, or it cannot
correct it [arxiv:2103.15297 §3.2].

## Variants

### Pinhole-prior depth proposal with a learned offset

The depth proposal is the focal length times the real object height, divided
by the height in pixels of the 2D box [arxiv:1904.01690 eq. 1]. The real height
comes from a predicted size, so the proposal is only as good as that size
estimate [arxiv:1904.01690 §3.1]. The refinement stage predicts an offset from
the proposal depth. The offset depends on rotation, dimensions and viewing
angles [arxiv:1904.01690 eq. 2]. The lateral position is not regressed. It is
read off the viewing ray through the estimated centre [arxiv:1904.01690 §3.2].
Card: [Monocular 3D Object Detection Leveraging Accurate Proposals and Shape Reconstruction](/papers/1904.01690).

The paper's depth-error ablation, on hard cars in val1 with no reconstruction
module, gives 1.22 m for proposal regression with full-image features, against
1.48 m for direct estimation with the same inputs [arxiv:1904.01690 tab. 6].
These are single-split, author-reported numbers.

### LiDAR box refinement with a boundary offset

The second stage takes a 3D proposal (x, y, z, w, l, h, θ) from a LiDAR
detector and reads the raw points inside an enlarged box around it. The points
are moved into the box frame, and each point gets six extra values: its
distance to the two faces on each axis [arxiv:2103.15297 §3.1] [arxiv:2103.15297 §3.3].
A small PointNet predicts a class score and corrected centre, size and heading.
The centre offset is divided by the proposal's w, l and h; the size target is a
log ratio; the heading target is the smaller of two residuals, so a 180° flip
is not penalised [arxiv:2103.15297 eq. 4] [arxiv:2103.15297 eq. 5]
[arxiv:2103.15297 eq. 7]. Card: [LiDAR R-CNN: An Efficient and Universal 3D Object Detector](/papers/2103.15297).

On the Waymo validation set, the paper reports vehicle 3D AP of 75.6 with
PointPillars proposals, against 72.1 for the PointPillars baseline
[arxiv:2103.15297 tab. 1]. Its size ablation puts the boundary-offset input at
75.6, against 74.1 for plain point refinement without the offset
[arxiv:2103.15297 tab. 4]. These are author-reported numbers on one dataset.

The paper also measures the problem it targets. After enlarging PointPillars
boxes by 1 m in length and width, 21.3% of the test boxes kept the same point
count [arxiv:2103.15297 §4.2]. The base detector is not re-trained
[arxiv:2103.15297 §1].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Calibrated pinhole camera and a size prior you trust for the class | Pinhole proposal, then a learned offset | The proposal follows from focal length and height [arxiv:1904.01690 eq. 1]; the offset corrects it [arxiv:1904.01690 eq. 2] |
| Predicted height is poor for the class | Expect the proposal to inherit that error | The proposal depends on the height estimate [arxiv:1904.01690 §3.1]. The paper does not measure this separately (not stated) |
| Inference must not use LiDAR | Pinhole proposal variant | Inference takes the image, 2D boxes and calibration; LiDAR only builds training targets [arxiv:1904.01690 §3.3] |
| Only depth error matters, with no 3D box output | Proposal regression over direct estimation | Lower depth error at the same inputs [arxiv:1904.01690 tab. 6] |
| LiDAR points are available and a 3D detector already gives boxes | LiDAR box refinement with a boundary offset | Author-reported gain over the base detector on Waymo vehicles [arxiv:2103.15297 tab. 1] |
| Several classes share one model | Boundary offset, not size normalisation or anchors | In the three-class run, the offset gives 73.4 / 70.6 / 66.8 AP (vehicle, pedestrian, cyclist), against 72.7 / 69.9 / 64.4 for size normalisation [arxiv:2103.15297 tab. 5] |
| Many proposals per frame and a tight latency budget | LiDAR box refinement | For 128 proposals the paper reports 4.5 ms and 0.5 M parameters, against 19 ms and 3.5 M for a voxel second stage [arxiv:2103.15297 tab. 9]. The hardware for these timings is not stated on the card |
| Pedestrians or cyclists in the monocular setting | Use with care | Results exist, but the paper reports no published comparison for these classes on the test server [arxiv:1904.01690 tab. 4] |
| Pedestrians or cyclists in the LiDAR setting | Check the class-by-class numbers | The multi-class validation table shows no gain for pedestrians with PointPillars proposals and a gain of 2.4 AP for cyclists, against 1.8 for vehicles [arxiv:2103.15297 tab. 2] |

Neither paper compares region-proposal or anchor-based proposals as
alternatives to a learned refinement. Add cards before using this page to
choose between them.

## Implementation sketch

The first sketch follows the monocular depth path only. The paper's full
pipeline also predicts the vertical and lateral position and the 3D shape
[arxiv:1904.01690 §3.1] [arxiv:1904.01690 §3.3].

```pseudocode
# Depth proposal from the pinhole relation  [arxiv:1904.01690 eq. 1]
# Inputs: focal length f in pixels, predicted real height H in metres,
# projected 2D box height h in pixels
def propose_depth(f, H, h):
    # Depth is focal length times real height over projected height
    return f * H / h

# Learned correction to the proposal  [arxiv:1904.01690 §3.2] [arxiv:1904.01690 eq. 2]
def refine_depth(z_proposal, features):
    # Offset depends on rotation, dimensions and viewing angles
    s = offset_head(features)
    return z_proposal + s
```

The second sketch covers the LiDAR box path. The points are used in the box
frame, and the targets are normalised by the proposal's size
[arxiv:2103.15297 §3.1] [arxiv:2103.15297 §3.3].

```pseudocode
# Refine one 3D proposal  [arxiv:2103.15297 §3.1] [arxiv:2103.15297 §3.3]
# Input:  points (N, 3); proposal (x, y, z, w, l, h, θ)
# Output: class score, corrected centre, size and heading

def refine_box(points, proposal):
    # Keep points inside the proposal's width and length, enlarged  [§3.1]
    local = to_box_frame(points_in_enlarged_bev(points, proposal), proposal)
    # Boundary offset: distance to the two faces on each axis  [§3.3]
    half = proposal.size / 2
    features = concatenate(local, half - local, half + local)
    # Fixed number of points per proposal  [§4.3]
    pooled = max_over_points(shared_mlp(sample_points(features)))
    hidden = fully_connected(pooled)
    return class_head(hidden), centre_head(hidden), size_head(hidden), heading_head(hidden)
```

## Contradictions

- **Lateral position (monocular).** The paper says the lateral coordinate is
  recovered from the viewing ray and not regressed [arxiv:1904.01690 §3.2]. The
  official code recomputes it at inference from the predicted box, a step the
  paper does not describe (see the monocular card's Paper vs code section).
  Which one the reported numbers use is not resolved.
- **Vertical position (monocular).** The paper places the proposal at the
  re-projected box centre [arxiv:1904.01690 §3.1]. The official code subtracts
  a fixed offset for cars, which the paper does not describe (see the
  monocular card's Paper vs code section).
- **Offset versus direct regression.** The monocular paper argues an offset from
  a proposal is more stable than direct regression [arxiv:1904.01690 §3.2]
  [arxiv:1904.01690 §5.2]. The LiDAR paper does not test offset against direct
  regression. Its ablations vary the input encoding and which targets are
  refined [arxiv:2103.15297 tab. 4] [arxiv:2103.15297 tab. 8]. The two results
  measure different things, depth error from images and AP from point clouds,
  so they cannot be ranked against each other.

## Papers

- [Monocular 3D Object Detection Leveraging Accurate Proposals and Shape Reconstruction](/papers/1904.01690): a pinhole depth proposal, refined by a learned offset, with a per-object point cloud trained from LiDAR-derived targets.
- [LiDAR R-CNN: An Efficient and Universal 3D Object Detector](/papers/2103.15297): a PointNet second stage that refines boxes from any LiDAR detector, with boundary-offset features that let it see proposal size.