LaneRCNN: Distributed Representations for Graph-Centric Motion Forecasting

In plain words

Self-driving cars need to guess where nearby cars and people will go next. Many earlier systems squeeze everything they know about one moving object, its recent path and the road around it, into a single list of numbers. The authors argue that this loses too much of the road's shape.

LaneRCNN instead gives each object its own small graph, built from the lanes that object could plausibly use. Each piece of lane is a node, and the links between nodes follow the road: what comes before, what comes after, and what sits to the left or right. The object's recent movement is stored on those nodes.

The per-object graphs are then placed onto one shared map graph, so nearby objects can pass information to each other along the lanes. The model scores which lane pieces are likely end points of the future path, turns each chosen end point into a smooth curve, and then refines that curve. On the Argoverse forecasting benchmark, the authors report a first place on the leaderboard.

TL;DR

• Predicts 3 s of future positions for every actor from 2 s of history and an HD map, on Argoverse §4.1.
• Each actor gets a LaneRoI: a lane graph around it plus node features that encode its motion §3.2.
• Actors interact on one global lane graph; the authors report that this beats fully connected attention and GNN interaction at equal depth §4.3.
• Output is goal-based: one score and offset per lane node, then a quadratic Bezier proposal and a learned Frenet refinement §3.4 app. A.
• Author-reported: miss rate 12.3 at K=6 on the Argoverse leaderboard, the first rank at submission tab. 1 §4.2.

Problem

Forecasting actor motion is hard because each actor's intention is hidden and its future depends on other actors and on the map §1. The authors list three limits of the common design, which encodes each actor as one feature vector §1 fig. 1:

1. A single vector has to summarise a large region; fast actors can cover about a hundred metres in five seconds §1.
2. A fully connected interaction graph among actors ignores map structure: two nearby cars on opposite lanes barely interact §1.
3. A plain regression head does not use lane information, so it sometimes predicts trajectories that leave the road §1.

The method assumes an HD map with lanes and attributes such as turning lanes and lanes controlled by traffic lights §3.1.

Method

The model predicts future positions of all actors in a scene from their past centre positions in bird's-eye view and an HD map §3.1. The pipeline has four stages fig. 2.

1. Build a LaneRoI per actor §3.2 app. B. Collect the lanes that the actor can reach within the horizon and those it came from within the history. Sample lane segments of 1 m along the centre lines app. B tab. 3. Connect segments by four relations: predecessor, successor, left neighbour and right neighbour §3.2. Each node embedding carries geometry (centre, orientation, curvature), binary semantic flags such as turning lane or red-light control, and the actor's recent 2D displacements expressed relative to that segment. If a segment is more than 5 m from the actor, its actor-motion part is set to zero §3.2.

2. Encode each LaneRoI with shared weights §3.3. The main operator is the lane convolution eq. 1. For each relation r and hop count n in {1, 2, 4, 8, 16, 32}, it sums features of nodes reachable in n steps through r, each with its own weight. A lane pooling operator eq. 2 interpolates features at arbitrary points, weighting neighbours by relative pose through a small MLP. A shortcut layer pools features along the actor's past path, passes them through a 1D convolution, and adds the result to every node §3.3 app. C.

3. Interact on a global lane graph §3.3. All LaneRoI node features are pooled onto one global lane graph. Four lane convolution layers run on that graph. Their output is pooled back to each LaneRoI and added as a skip connection, followed by further encoder layers app. C.

4. Decode goals and trajectories §3.4 app. A. A two-layer MLP per node outputs a goal probability and four values: the offsets to the goal in x and y, and the sine and cosine of the heading difference. The top K = 6 goals are kept. For each goal, a quadratic Bezier curve connects the actor's current pose to the goal pose, and a constant-acceleration profile places the waypoints on it. A learned head then predicts a correction for each future time step in the Frenet frame of that proposal.

Training uses a classification loss over nodes, a regression loss over positive nodes and a refinement loss, combined as L = L_cls + α·L_reg + β·L_refine §3.5. The classification term is binary cross entropy with online hard example mining app. C.

Key equations, as printed in the paper (numbered in the paper as eq. 1 and eq. 2):

\mathbf{F}_i \leftarrow \Psi\left(\mathbf{F}_i\mathbf{W} + \sum_{r,n} \mathrm{bool}\left(\mathcal{E}_i^n(r)\right)\mathbf{F}_i\mathbf{W}_{n,r}\right) eq. 1

where W and W_{n,r} are learnable, Ψ is LayerNorm followed by ReLU, and bool sets non-zero entries to one §3.3.

f_{\mathbf{v}} = \mathcal{M}_b\left(\sum_{k \in \mathcal{N}} \mathcal{M}_a\left(\left[f_k, \Delta_{\mathbf{v}k}\right]\right)\right) eq. 2

where [·] is concatenation, Δ is the relative pose between point v and segment k, N is the set of nearby nodes, and M_a and M_b are two-layer MLPs §3.3.

# LaneRCNN forward pass  [§3.2] [§3.3] [§3.4]
# Input:  past (L, 2) per actor; lane graph of the map
# Output: per actor, K goal scores and K trajectories (T, 2)

def predict(past, map_graph):
    # One LaneRoI per actor, from the lanes it can reach  [§3.2] [app. B]
    rois = [build_laneroi(actor, map_graph) for actor in past]

    # Encode every LaneRoI with shared weights  [§3.3] [app. C]
    for roi in rois:
        roi.features = encode_laneroi(roi)

    # Pool all LaneRoIs onto one global lane graph  [§3.3]
    global_graph = pool_to_global(rois)
    global_graph.features = lane_conv_layers(global_graph)

    # Pool global features back into each LaneRoI, as a skip  [§3.3]
    for roi in rois:
        roi.features = roi.features + distribute(global_graph, roi)
        roi.features = lane_conv_layers_after(roi)

    # Score each node as a goal and regress its offset  [§3.4]
    for roi in rois:
        goal_score, goal_offset = goal_head(roi.features)

    # Keep the top K goals and build a trajectory for each  [§3.4] [app. A]
    return [propose_trajectories(goal_score, goal_offset, roi) for roi in rois]

# Trajectory proposal from one goal  [app. A]
# Input:  start pose (x, y, dx, dy), goal pose (x, y, dx, dy),
#         initial speed v, horizon T
# Output: waypoints at the future time steps, (N, 2)

def propose_trajectory(start, goal, v, horizon):
    # Quadratic Bezier with the start and goal positions and the
    # tangent directions at both ends  [app. A]
    curve = fit_quadratic_bezier(start, goal)
    # Arc length from start to goal along that curve  [app. A]
    distance = curve_length(curve)
    # Constant acceleration that covers the arc length in T  [app. A]
    acceleration = 2.0 * (distance - v * horizon) / horizon ** 2
    waypoints = []
    for t in future_times:
        # Distance travelled along the curve at time t  [app. A]
        s = v * t + 0.5 * acceleration * t ** 2
        waypoints.append(point_on_curve(curve, s))
    return waypoints

Hyperparameters

Results

Author-reported. Setup: Argoverse Motion Forecasting benchmark, 3 s future from 2 s history at 10 Hz §4.1. Table 1 compares on the leaderboard, whose test labels are withheld; the paper submitted once, with a snapshot dated 12 Nov 2020 §4.2 tab. 1. The official ranking metric is miss rate (MR) at K=6 §4.1.

Lower is better for all metrics tab. 1. The comparison covers the top leaderboard entries and the official Argoverse baselines §4.2. The Poly method is unpublished, so the authors could not compare it qualitatively §4.2.

Ablations are on the validation set tab. 2. Upper half: encoder variants. Lower half: interaction variants. The shaded row is the final model.

Lane sampling and regression tab. 3 (validation):

Output parameterisation tab. 4 (validation; the table's own column names are kept, with minADE at K=1 and K=6):

Other reported results: with only 1% of the training data, the authors report a miss rate of about 20% for the lane-based output, while a regression head needs much more data app. D fig. 6. Qualitative examples cover curved lanes, lane changes, intersections and overtaking §4.4 fig. 5 fig. 7.

Paper vs code

No official code.

The paper gives no code link. The bibliography points only to the Argoverse leaderboard §4.2. Searches on GitHub, using the repository search API with the queries "LaneRCNN", "laneroi", "lane rcnn", "LaneRCNN org:uber-research" and "lane graph motion forecasting argoverse", returned no repository by the authors or their lab. Web search was unavailable in this run, so no other search engine was checked.

Pitfalls

• Layer counts are described two ways. The main text gives four lane-convolution layers per stage with shortcuts "after any layer" §3.3. The appendix gives two layers, a shortcut, two layers and a shortcut for the encoder app. C. Both give four layers per block. Implement the appendix version, which states the placement.
• Pooling radius is only partly stated. The 2 m neighbourhood applies to the global graph app. C. The radius used when pooling inside a LaneRoI is "a certain threshold" with no value §3.3.
• Clamp is easy to misread. Segments more than 5 m from the actor have their actor-motion part set to zero, not dropped from the graph §3.2.
• Lane sampling matters a lot. Keeping the dataset's own segment labels, which vary in length, gives much worse results than uniform 1 m segments: K=6 MR 24.0 against 8.2 tab. 3 app. D. The paper says this is because varying lengths add variance.
• Bezier boundary condition is printed as a ratio. The paper writes the tangent condition as x'(0)/x'(1) = dx0/dxT, a ratio of derivatives at the two ends. It does not say how the start and end tangents are scaled, and there is no code to resolve it app. A.
• Acceleration formula is unnumbered. a = 2(s - vT)/T² and s(t) = vt + ½at² come from the appendix, where s is the arc length along the curve and v the initial speed app. A. The paper gives no rule for the case where the acceleration is negative or too large.
• Frenet residual sign. The sign of d gives the side: positive means left of the curve app. A. The procedure that projects ground truth onto each proposal to build refinement targets is described only briefly §3.5.
• Duplicate-goal removal threshold is not stated. The paper drops the lower-confidence goal when two are "too close" §3.4 footnote.
• Classification sampling is detailed but thin. Positives are the node closest to the ground-truth goal. Negatives are nodes more than 6 m away. One quarter of them are sampled, and the hardest 100 are kept per sample. Nodes in between are ignored. The paper does not state how "hardest" is measured app. C.
• Regression loss has an unstated shape. The smooth-L1 transition point is not given, and the regression loss uses positive nodes only §3.5.
• Loss weights are only in the appendix. α = 0.5 and β = 0.2 appear only in app. C, not in the main loss definition §3.5.
• Two result sets, different splits. The leaderboard numbers (test) and the ablation numbers (validation) differ widely: the final model is K=6 MR 12.3 on the leaderboard and 8.2 on validation. The paper does not reconcile them. Compare ablation rows only with each other.
• Single submission, competitor details missing. The leaderboard is a single submission, and the Poly method is unpublished §4.2.
• Cost is not reported. The paper does not report training time, hardware, memory or inference latency. Lane convolution uses matrix powers of the adjacency matrix up to hop 32, for each relation §3.3, which can be costly for large maps; the paper does not measure it.
• Augmentation at test time is not stated. Only training uses random rotation §4.1.

Lineage

• Builds on: lane convolution and lane pooling from LaneGCN §3.3; dilated convolution for the multi-hop idea §3.3; online hard example mining for the classification loss app. C.
• Compared against: TNT, Jean and Poly on the Argoverse leaderboard tab. 1; transformer-style attention and GNN interaction modules with the same backbone §4.3 tab. 2. The authors note that TNT uses a similar output parameterisation but that LaneRCNN performs better on all metrics §4.2.