Joint 3D Proposal Generation and Object Detection from View Aggregation

In plain words

Self-driving cars need to know where other vehicles and people are in 3D, including which way they face. This paper's detector, called AVOD, uses two sensors: a laser scanner (LiDAR), which gives depth, and a camera, which gives colour and detail. The laser points are flattened into a top-down picture, and the camera image is processed on its own. Both pictures are turned into feature maps that keep fine detail.

A first stage looks at those maps and proposes many candidate 3D boxes. It checks both views for each candidate and keeps the ones that look like objects. A second stage then tightens each box, decides what the object is, and works out which way it points. The authors report that the method is accurate on road scenes and fast enough for a car to run in real time, and that it needs little memory.

TL;DR

• AVOD is a two-stage 3D detector for road scenes. Both stages share one feature extractor per view: a bird's-eye-view (BEV) LiDAR map and the RGB image §3 §3.2.
• The region proposal network (RPN) fuses full-resolution crops from both views and predicts an objectness score and six anchor offsets per anchor §3.3 fig. 2.
• The second stage predicts a compact 4-corner box with two heights, plus an orientation angle vector (cos θ, sin θ) that picks one of four box orientations §3.4.
• Author-reported results on KITTI are in the Results section. Two networks are trained: one for cars, one for pedestrians and cyclists §3.5 §4.

Problem

Detecting objects as oriented 3D boxes is harder than 2D detection. Small or distant objects occupy only a few cells of a coarse BEV map, and the usual downsampled feature maps lose them §1 §2. Earlier 3D RPNs such as MV3D reduce the BEV resolution too much for small classes and encode boxes in a way that makes orientation ambiguous §2 §3.4. Monocular methods depend on 2D detectors, so a missed 2D detection becomes a missed 3D detection. The paper expects this to be a problem in extreme weather and lighting §2.

The authors aim for high proposal recall for small classes, a box encoding that keeps the physical box constraints, and a stage-two orientation estimate that is not confused by ±π ambiguity §1 §3.4.

Method

Overview §3 fig. 2. Two identical feature extractors, one for the BEV map and one for the image, produce feature maps. The RPN and the second-stage detector both consume the feature maps. Fig. 2 colours the feature extractors, RPN and second stage differently.

BEV input §3.1. A voxel grid of the LiDAR points is turned into a six-channel map at 0.1 m resolution. The point cloud is cropped to x in [−40, 40] m and z in [0, 70] m, the part inside the camera field of view. Channels 1–5 hold the maximum point height in each cell for five equal slices. Channel 6 is a density value, min(1, log(N+1)/log 16), where N is the number of points in the cell.

Feature extractor §3.2 fig. 3. The encoder follows VGG-16 with half the channels and stops after conv-4. It reduces resolution by 8×, so a pedestrian of about 0.8 × 0.6 m (roughly 8 × 6 cells in the BEV map) covers less than one cell in the output map. The decoder upsamples step by step with transposed convolutions. At each of the three scales it concatenates the encoder feature of the same size and fuses them with a 3×3 convolution. The output is at full input resolution and has depth D̃ = 32 §3.4 pyramid_cars_example.config.

Anchors §3.3. Anchors are axis-aligned 3D boxes (centre and size, no orientation). Their BEV centres lie on a 0.5 m grid. Their height offset follows the sensor height above the ground plane. Their sizes come from clustering the training boxes of each class. Anchors with no LiDAR points in BEV are removed, leaving about 80–100K anchors per frame §3.3 pyramid_cars_example.config.

Multiview crop and fusion §3.3 fig. 2. Each anchor is projected into the BEV map and the image. The regions are cropped from each feature map and bilinearly resized to 3×3. Before cropping, a 1×1 convolution reduces each view's features, which cuts the memory needed for crops by a factor of D̃. The two crops are fused by element-wise mean, so the RPN gets one set of features per anchor §3.3.

RPN heads §3.3. Two branches of fully connected layers, each of width 256, sit on the fused crops. One outputs objectness (object vs. background). The other outputs six offsets per anchor: centre (Δt_x, Δt_y, Δt_z) and size (Δd_x, Δd_y, Δd_z). Objectness uses cross-entropy. Offsets use smooth L1, and background anchors are excluded from the regression loss. An anchor counts as background if its BEV IoU with every ground-truth box is below 0.3 for cars. It counts as an object above 0.5 for cars and above 0.45 for pedestrians and cyclists §3.3.

Proposals §3.3. Regressed anchors are reduced by BEV non-maximum suppression (NMS) at IoU 0.8. During training the top 1024 are kept. At inference 300 are kept for cars and 1024 for pedestrians and cyclists §3.3.

Box encoding §3.4 fig. 4. Instead of eight corners (24 values), the box is coded with four BEV corners and two heights, the top and bottom offsets from the ground plane. That gives a 10-value target: four x offsets, four z offsets, and two height offsets. Each proposal corner is matched to the closest ground-truth corner §3.4. The paper argues that this avoids redundant free parameters in the eight-corner form §3.4.

Orientation §3.4 fig. 1. The network regresses an angle vector (cos θ, sin θ), which handles angle wrap-around. A four-corner box has four possible orientations. The paper picks the one whose direction is closest to the regressed angle vector §3.4. The pipeline is written in the code as a post-processing step (see Paper vs code).

Second stage §3.4 tab. 1–2. Proposals whose BEV IoU with ground truth is at least 0.65 for cars, or 0.55 for pedestrians and cyclists, count as positives for the regression loss §3.4. Crops from both views are resized to 7×7 from the D̃ = 32 map, fused by mean, and passed through three fully connected layers of width 2048. The outputs are box regression, orientation vector and class scores. BEV NMS at IoU 0.01 removes duplicates §3.4.

Training §3.5. Two networks are trained: one for cars, one shared by pedestrians and cyclists. RPN and second stage are trained jointly end to end. Each mini-batch holds one image with 512 RPN anchors and 1024 second-stage proposals. Training uses Adam with an initial learning rate of 1e-4, decayed by a factor of 0.8 every 30K iterations, for 120K iterations §3.5.

Losses. The paper combines smooth L1 for box and orientation regression with cross-entropy for classification. It gives no loss weights and no normalisation rule §3.3 §3.4. The code uses weights 1.0 (classification), 5.0 (offsets) and 1.0 (angle), so do not assume unit weights pyramid_cars_example.config.

# AVOD forward pass, one frame, car network  [§3] [§3.1]-[§3.4]
# Input:  lidar points, RGB image, ground plane
# Output: oriented 3D boxes with class scores

def detect(points, image, ground_plane):
    # Six-channel BEV map: five height slices plus density  [§3.1]
    bev_map = make_bev_map(points, ground_plane)     # (H, W, 6)
    # Full-resolution feature maps, one extractor per view  [§3.2]
    bev_feats = bev_extractor(bev_map)               # (H, W, 32)
    img_feats = img_extractor(image)                 # (H', W', 32)
    # 1x1 bottleneck per view, before cropping  [§3.3]
    bev_small = conv1x1(bev_feats)                   # (H, W, 1)
    img_small = conv1x1(img_feats)                   # (H', W', 1)
    # Anchors on a 0.5 m BEV grid, empty ones removed  [§3.3]
    anchors = make_anchors(points, stride=0.5)
    # Crop each anchor from both views to 3x3, then mean-fuse  [§3.3]
    bev_crops = crop_resize(bev_small, anchors, 3)
    img_crops = crop_resize(img_small, anchors, 3)
    fused = mean(bev_crops, img_crops)
    # RPN: objectness and six offsets per anchor  [§3.3]
    objectness, offsets = rpn_heads(fused)
    # Decode, BEV NMS at IoU 0.8, keep 300 (car) or 1024  [§3.3]
    proposals = bev_nms(decode(anchors, offsets), objectness,
                        iou=0.8, keep=300)
    # Stage two: crop proposals from both maps to 7x7, mean-fuse  [§3.4]
    bev_crops = crop_resize(bev_feats, proposals, 7)
    img_crops = crop_resize(img_feats, proposals, 7)
    fused = mean(bev_crops, img_crops)
    # Three FC layers of 2048; outputs box, angle vector, class  [§3.4]
    box_4c, angle_vec, class_scores = stage_two(fused)
    # Rebuild a 3D box and pick the orientation nearest the angle vector  [§3.4]
    boxes = resolve_orientation(box_4c, angle_vec)   # see Paper vs code
    # BEV NMS at IoU 0.01  [§3.4]
    return bev_nms(boxes, class_scores, iou=0.01)

# Training loss, one image  [§3.3] [§3.4] [§3.5]
# Positive thresholds: RPN above 0.5, stage two at least 0.65 (cars)  [§3.3]

def training_loss(rpn_out, head_out, targets):
    # RPN objectness: cross-entropy on 512 sampled anchors  [§3.3] [§3.5]
    rpn_cls = cross_entropy(rpn_out.objectness, targets.rpn_labels)
    # RPN offsets: smooth L1 on object anchors only  [§3.3]
    rpn_reg = smooth_l1(rpn_out.offsets, targets.rpn_offsets)
    # Stage two: cross-entropy on 1024 proposals  [§3.4] [§3.5]
    head_cls = cross_entropy(head_out.scores, targets.head_labels)
    # Stage two: smooth L1 on box offsets and on the angle vector  [§3.4]
    head_reg = smooth_l1(head_out.offsets, targets.head_offsets)
    head_ang = smooth_l1(head_out.angle, targets.angle_vectors)
    # Paper gives no weights; code uses 1.0 / 5.0 / 1.0 (see Paper vs code)
    return rpn_cls + rpn_reg + head_cls + head_reg + head_ang

Hyperparameters

Every value below comes from the paper unless the "Where" column names the code. "not stated" means neither the paper nor the code sets it in the cited place.

Results

Author-reported. All numbers are the paper's own, measured on KITTI. Tables and figures are numbered here in order of appearance in the LaTeX source.

Validation set, cars, 3D AP and AHS at 0.7 3D IoU tab. 1. Deep3DBox and MV3D are compared using their publicly released detections. AHS is the authors' heading-similarity metric §4.

KITTI test set, 3D and BEV AP (%) tab. 2. Test results come from the KITTI evaluation server. The server does not compute AHS §4.

Ablations on validation, moderate difficulty tab. 3 §4.1. Cars use 0.7 3D IoU; pedestrians and cyclists use 0.5. Parameter and FLOP columns are as printed (see Pitfalls).

Other author-reported claims §4 fig. 5 fig. 7.

• Proposal recall (3D IoU 0.5, validation): the Feature Pyramid RPN reaches 86% for cars with 10 proposals per frame. The best recall of 3DOP and Mono3D for cars is 73.87% and 65.74%. For cars, the RPN reaches 91% with 50 proposals, where MV3D reported needing 300.
• Fusing image and BEV features in the RPN improves pedestrian and cyclist recall at 1024 proposals over BEV-only, by 10.1% and 8.6% (without the pyramid extractor) §4 fig. 5.
• Runtime on a TITAN Xp GPU: about 0.1 s per frame, of which about 20 ms is preprocessing and about 80 ms inference. The authors report 231.263 billion FLOPs per frame and about 2 GB of GPU memory at inference §4.
• Qualitative results: fig. 6 (proposals and detections), fig. 7 (AVOD vs MV3D vs Deep3DBox against ground truth).

Paper vs code

Official code: github.com/kujason/avod, owned by the first author. Commit b2d32f6ddd5007c12afe37760fbcff993816da39 was checked; the repository is MIT licensed. Discrepancies, each checked against the code:

• Height slices. The paper gives slices over [0, 2.5] m §3.1. The code slices from −0.2 to 2.3 m relative to the ground plane, five slices of 0.5 m each pyramid_cars_example.config bev_slices.py bev_slices.py. The density channel uses the same band bev_slices.py.
• Bottleneck width. The paper says the 1×1 reduction divides the crop memory by D̃ §3.3. The code reduces each view to one channel before cropping rpn_model.py rpn_model.py. Stage two uses the full D̃ = 32 map avod_model.py.
• Mean fusion with path drop. The paper says mean fusion. The code divides by the number of branches that survive path drop (2 when both are kept) rpn_model.py rpn_model.py. The paper does not describe path drop.
• Orientation resolution. The paper picks the closest of four candidate orientations §3.4. The code does this after NMS, in the evaluator, not in the network. It compares the box yaw with the predicted angle. It rotates by 90° when the difference lies between π/4 and 3π/4 (in absolute value), and it flips by π when the absolute difference is at least 3π/4 evaluator.py evaluator.py. Those thresholds are not stated in the paper.
• Four-corner frame. The code first rounds the yaw to the nearest multiple of 90° and encodes corners in that frame box_4c_encoder.py. The paper does not describe this.
• Box side selection when decoding. When converting a 4-corner box back to a 3D box, the code keeps whichever of the two opposite-edge pairs has the larger magnitude, and uses its centroid, length, width and yaw box_4c_encoder.py. The paper does not describe this.
• Loss weights and normalisation. The paper gives no loss weights §3.3 §3.4. The code uses 1.0 / 5.0 / 1.0 pyramid_cars_example.config. RPN objectness is normalised by the mini-batch size and RPN regression by the number of positives rpn_model.py. Stage-two classification is normalised by mini-batch size, and regression and angle losses by the number of positives avod_loss_builder.py avod_loss_builder.py.
• Total loss. The code adds the RPN loss and the stage-two loss with no extra weight avod_model.py.
• Undocumented regularisers. The code uses label smoothing of 0.001 pyramid_cars_example.config, dropout with keep probability 0.5 pyramid_cars_example.config, batch normalisation after every extractor convolution bev_vgg_pyramid.py, and L2 weight decay. The paper mentions none of these.
• Input padding. The BEV extractor pads 4 rows at the top before pooling, to make 700 rows divisible by 8 bev_vgg_pyramid.py.
• Output cap. The stage-two NMS keeps at most 100 boxes pyramid_cars_example.config. The paper does not state a cap.
• Cluster counts. The code uses 2 anchor clusters for cars and 1 each for pedestrians and cyclists pyramid_cars_example.config. The paper does not give a count.
• Test-time sampling. The people config evaluates all samples, while the car config does not pyramid_people_example.config pyramid_cars_example.config.

Everything else checked matches: the BEV density formula §3.1 bev_generator.py, the 0.5 m anchor stride, the RPN and stage-two IoU thresholds, NMS values, proposal counts, both 256 and 2048 FC widths, Adam with learning rate 1e-4 decayed by 0.8 every 30K iterations, and 120K iterations.

Pitfalls

• Height band. The paper's "[0, 2.5] m" is ambiguous (height above what?). The code uses −0.2 to 2.3 m above the estimated ground plane. Use the code's band unless you have checked otherwise.
• Table III parameter column is internally inconsistent. The base row lists 38,073,528 parameters, which is the figure the text gives for the final model. The Feature Pyramid row is a delta of −21,391,104 from base. Those two do not reconcile. The FLOP column does: 186.28 B + 44.98 B = 231.26 B, the figure the text gives. Do not rely on the parameter column.
• Table III base network. The paper says the base uses MV3D's extractor §4.1. Read the table rows as deltas where the caption says so, and do not take absolute numbers from them without checking.
• Ground-plane dependence. The 4-corner encoding and the height channels depend on a ground-plane estimate, and the sensor height sets the anchor heights §3.1 §3.3 §3.4. The paper does not say how the plane is obtained. The code reads it per sample as a model input rpn_model.py. Check how your own data pipeline supplies it.
• Orientation ambiguity needs post-processing. Without the evaluator's rotation and flip rules, a 4-corner box can carry the wrong heading by ±90° or 180°. The network output alone does not give the final yaw.
• Anchor count and memory. The paper says 80–100K non-empty anchors per frame, and the 1×1 bottleneck is what keeps crops affordable. Its own estimate is about 5 GB for 7×7 crops of 256-channel features over 100K anchors §3.3. Keep the bottleneck.
• Pedestrian and cyclist shared network. The paper trains one network for both classes §3.5. Cyclist results are weaker than F-PointNet's on the test set, which the authors attribute to fewer cyclist instances in KITTI §4.
• Framework. The official code is TensorFlow 1.x (it uses tf.contrib.slim), so it will not run unmodified on TensorFlow 2 bev_vgg_pyramid.py.
• Speed claims. The runtime figures are for one TITAN Xp and the stated preprocessing. Compare with other methods on the same hardware before drawing conclusions.
• Test-set comparisons. Most comparisons use numbers from the KITTI server or published detections, not retrained baselines §4.

Lineage

• Builds on: region proposal networks from Faster R-CNN §2; feature pyramid networks §1 §3.2; the BEV and image two-view design of MV3D, with changes to the RPN and encoding §2 §3.3 §3.4.
• Ideas borrowed from other work: 1×1 bottleneck convolutions inspired by SqueezeNet §3.3; crop-and-resize of features as in the cited crop-and-resize operation §3.3.
• Compared against: MV3D, Deep3DBox §4 tab. 1; 3DOP and Mono3D for proposals §4; VoxelNet and F-PointNet on the test set tab. 2.