Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline

In plain words

A self-driving system that learns from human or expert driving usually learns one of two things. Either it predicts a short path, a list of points the car should pass through, and a separate controller turns that path into steering and pedal commands. Or it predicts the steering and pedal commands directly. Each choice has a weak spot. Path-based systems depend on the controller following the path well, and the controller is hard to tune in tight turns. Direct systems react late and look at one moment at a time.

The paper puts both outputs into one network that shares the camera features. The path branch predicts the route ahead. The control branch predicts commands for the next few moments, and it looks at the camera image with attention that the path branch guides, so it knows which parts of the scene matter at each future moment. At the end, the two outputs are blended. The blend favours whichever output is usually better in the current situation, and the authors use whether the car is turning as that test.

The system uses one front camera. The authors report that it ranked first on a public driving benchmark in a simulator, ahead of entries that used several cameras and a laser scanner. Those numbers are the authors' own and come from one simulator, so treat them as a claim to check, not a settled ranking.

TL;DR

The paper trains one network with a trajectory branch (waypoints) and a multi-step control branch (steer, throttle, brake for the next K = 4 steps) that share an image encoder §3.2 fig. 2. The control branch gets per-step attention over the image grid from the trajectory branch §3.2.2 eq. 2 eq. 3. Both branches imitate a Roach reinforcement-learning expert §3.1. At test time the two outputs are blended with weight 0.3, using a turning rule §3.4 alg. 1. Author-reported: 75.137 driving score for the ensemble and 69.714 for the single model on the CARLA leaderboard tab. 1. The official repository's LICENSE is Apache-2.0, but the paper's supplement says CC BY-NC 4.0, and the repository's code differs from the paper in several places. Read Paper vs code before reproducing.

Problem

End-to-end driving maps sensor input to either a planned trajectory or to control actions §1. Trajectory outputs look further ahead and can be combined with other modules, but they need a low-level controller to be followed, and simple PID controllers do poorly in big turns §1. Direct control avoids that step but acts on the current moment only, so it can react late to obstacles and produce less stable actions §1 fig. 1.

The authors find no clear winner between the two output forms across scenarios §1. Training two separate models and averaging them doubles the size of the model and ignores the links between the tasks §1.

Evaluation is closed-loop in CARLA on routes with navigation commands and scenarios such as obstacle avoidance and unsignalised crossings §4.1. The main metric, the driving score, is route completion multiplied by an infraction score §4.1.

Method

The model has one image and measurement encoder and two output branches §3.2. The trajectory branch predicts K waypoints. The control branch predicts a probability distribution over controls for the current step and for each of K future steps §3.2.2. Both branches are trained by imitation of an expert (Roach) and combined at test time by a situation rule §3.4.

Expert and supervision (§3.1)

The expert is Roach, an RL policy trained with privileged information (roads, lanes, routes, vehicles, pedestrians, traffic lights and stops) rendered as a bird's-eye-view image §3.1. The student is trained on expert actions and on expert latent features, with a value head that predicts the expert's expected return §3.1 §3.3. The authors describe the trajectory-branch target as ground-truth expert waypoints, which they treat as a planning task rather than pure behaviour cloning §3.1.

The behaviour-cloning objective is written as expected loss over expert state-action pairs eq. 1.

\arg\min_\theta \, \mathbb{E}_{(x, a^*) \sim D}\big[\mathcal{L}(a^*, \pi_\theta(x))\big] \quad [\text{eq. 1}]

Encoders (§3.2)

• An image encoder, a ResNet-34 pretrained on ImageNet, produces a feature grid F §3.2 [supp. Implementation Details]. The input is one front image of 900 x 256 pixels with a 100 degree field of view [supp. Implementation Details].
• A measurement encoder (MLP) takes the speed, a discrete navigation command and the target point §3.2.
• The shared features go to both branches §3.2 fig. 2.

Trajectory branch (§3.2.1)

• The grid is average-pooled and concatenated with the measurement feature to form j_traj §3.2.1.
• A GRU generates the K waypoints one after another from j_traj §3.2.1.
• Two PID controllers turn the waypoints into throttle, brake and steer §3.2.1. The longitudinal PID speed comes from the distances between consecutive waypoints, and the lateral PID steers toward their direction §3.2.1.

Multi-step control branch (§3.2.2)

The control branch predicts a_t, a_{t+1}, ..., a_{t+K} from the current input §3.2.2. It has three parts:

1. Temporal module. A GRU takes the current feature and the current predicted action and updates the hidden state. The authors describe it as a coarse simulator of the scene and the ego vehicle §3.2.2.
2. Trajectory-guided attention. At each future step, an attention map over the grid is computed from the trajectory branch's hidden state and the control branch's hidden state eq. 2. The attended grid feature is combined with the control hidden state to form j_ctl_t eq. 3. The attention at the first step uses only the measurement feature §3.2.2 [supp. network table].
3. Shared policy head. The same head maps each j_ctl_t to a distribution over the action §3.2.2.

Actions are modelled as Beta distributions, and the loss is a KL divergence between the expert's Beta and the predicted Beta §3.3 eq. 5.

w_t = \mathrm{MLP}\big(\mathrm{Concat}[h^{traj}_t, h^{ctl}_t]\big), \quad w_t \in \mathbb{R}^{1 \times H \times W} \quad [\text{eq. 2}]

j^{ctl}_t = \mathrm{MLP}\big(\mathrm{Concat}[\mathrm{Sum}(\mathrm{Softmax}(w_t) \odot F),\ h^{ctl}_t]\big) \quad [\text{eq. 3}]

The step-by-step forward pass is in the pseudocode below.

# Trajectory-guided control, one forward pass  [§3.2]
# B: batch. K = 4 future steps. Grid is 8 x 29 for a 900 x 256 image.

def tcp_forward(image, speed, command, target_point):
    # Shared ResNet-34 encoder  [§3.2] [supp. Implementation Details]
    grid = resnet34_feature_map(image)           # (B, 512, 8, 29)
    grid = reshape(grid, (B, 512, 232))
    # Pooled image vector: average over the grid  [§3.2.1]
    # The code uses the 1000-d fc output here  [code: TCP/resnet.py#L244]
    pooled = mean(grid, axis=2)                  # (B, 512)
    # Measurement encoder: speed, command one-hot, target point  [§3.2]
    measure = measurement_mlp(speed, command, target_point)
    # Trajectory branch: GRU writes waypoints one by one  [§3.2.1]
    traj_feat = join_traj(concat(pooled, measure))   # (B, 256)
    wp_prev = zeros(B, 2)
    hidden = traj_feat
    for t in range(K):
        # The code also feeds the target point at every step
        # [code: TCP/model.py#L155-L160]
        hidden = gru_traj(concat(wp_prev, target_point), hidden)
        wp_prev = wp_prev + output_traj(hidden)  # residual step
        waypoints[t] = wp_prev
        traj_hidden[t] = hidden
    # Control branch, step 0: attention from measurements only  [§3.2.2]
    weights = softmax(init_att(measure))         # (B, 232)
    j_ctl = mlp_ctl(concat(sum(grid * weights, axis=2), measure))
    actions = [policy_head(j_ctl)]               # Beta params, step 0
    ctl_hidden = zeros(B, 256)
    for t in range(K):
        # Temporal module: feature and current action  [§3.2.2]
        # The code feeds Beta parameters, not a sample
        # [code: TCP/model.py#L185-L186]
        ctl_hidden = gru_ctl(concat(j_ctl, actions[-1]), ctl_hidden)
        # Trajectory-guided attention over the grid  [eq. 2]
        weights = softmax(mlp_att(concat(traj_hidden[t], ctl_hidden)))
        # Re-aggregate the grid for this step  [eq. 3]
        attended = sum(grid * weights, axis=2)
        j_ctl = mlp_merge(concat(attended, ctl_hidden))
        actions.append(policy_head(j_ctl))       # step t + 1
    return waypoints, actions                    # K and K + 1 outputs

Output fusion (§3.4)

The trajectory output goes through the PID controllers to become a control, and the control branch gives a second control. The two are mixed with a weight alpha in [0, 0.5] chosen by the current situation §3.4 alg. 1. The situation is "turning" or "not turning", so the rule gives the control branch more weight when the car is turning §3.4 [supp. Implementation Details].

# Situation-based fusion  [alg. 1] [§3.4]
# alpha in [0, 0.5]. Applied to steer, throttle and brake.

def fuse(control_branch, traj_pid, turning, alpha):
    # Trajectory-specialised situation: trajectory output keeps 1 - alpha
    if not turning:
        return alpha * control_branch + (1 - alpha) * traj_pid
    # Control-specialised situation: control output keeps 1 - alpha
    return alpha * traj_pid + (1 - alpha) * control_branch

Loss (§3.3)

The total loss is a weighted sum of the trajectory loss, the control loss and an auxiliary loss eq. 6. Each term is written out below.

\mathcal{L}_{traj} = \sum_{t=1}^{K} \|wp_t - \hat{wp}_t\|_1 + \lambda_F \, \mathcal{L}_F\big(j^{traj}_0, j^{Expert}_0\big) \quad [\text{eq. 4}]

\begin{aligned} \mathcal{L}_{ctl} = {} & \mathrm{KL}\big(\mathrm{Beta}(a_0) \,\|\, \mathrm{Beta}(\hat a_0)\big) + \frac{1}{K}\sum_{t=1}^{K} \mathrm{KL}\big(\mathrm{Beta}(a_t) \,\|\, \mathrm{Beta}(\hat a_t)\big) \\ & + \lambda_F \, \mathcal{L}_F\big(j^{ctl}_0, j^{Expert}_0\big) + \frac{1}{K}\sum_{t=1}^{K} \mathcal{L}_F\big(j^{ctl}_t, j^{Expert}_t\big) \end{aligned} \quad [\text{eq. 5}]

\mathcal{L} = \lambda_{traj} \mathcal{L}_{traj} + \lambda_{ctl} \mathcal{L}_{ctl} + \lambda_{aux} \mathcal{L}_{aux} \quad [\text{eq. 6}]

In words: the trajectory loss is an L1 distance on waypoints plus an L2 feature loss at the current step eq. 4. The control loss is a KL divergence between Beta distributions at the current step and averaged over the future steps, plus feature losses at the same steps eq. 5. The auxiliary loss is a weighted sum of an L1 speed loss and an L2 value loss, both from the image feature §3.3. Losses for future steps are averaged and then added to the current-step loss, because the action executed now is the main target §3.3.

# Training loss  [§3.3] [eq. 4] [eq. 5] [eq. 6]
# Targets: expert waypoints, expert Beta parameters per step, expert
# latent features and value (from Roach).

def tcp_loss(out, expert):
    # Trajectory L1 over waypoints, plus feature loss at step 0  [eq. 4]
    traj_loss = sum_t l1(out.wp[t], expert.wp[t])
    traj_loss = traj_loss + lam_f * mse(out.traj_feat_0, expert.feat_0)
    # Control KL at step 0, then averaged over future steps  [eq. 5]
    ctl_loss = kl(expert.beta[0], out.actions[0])
    ctl_loss = ctl_loss + mean_t kl(expert.beta[t], out.actions[t])
    # Feature losses at step 0 and averaged over future steps  [eq. 5]
    ctl_loss = ctl_loss + lam_f * mse(out.ctl_feat_0, expert.feat_0)
    ctl_loss = ctl_loss + mean_t mse(out.ctl_feat[t], expert.feat[t])
    # Auxiliary: L1 speed, L2 value  [§3.3]
    aux_loss = lam_speed * l1(out.speed, speed)
    aux_loss = aux_loss + lam_value * mse(out.value, value)
    return lam_traj * traj_loss + lam_ctl * ctl_loss + lam_aux * aux_loss

Hyperparameters

Values are from the supplement unless a code citation is given. Where the paper and code disagree, both are listed in Paper vs code.

Network structure, from [supp. network table]. Widths and activations are as printed there.

Results

Author-reported. Setup: CARLA 0.9.10.1 public leaderboard, accessed May 2022 tab. 1 [supp. Implementation Details]. The main model is trained on all 420K samples for the leaderboard; ablations train on 189K samples from four towns §4.1. Ablation tests use four routes in Town02 and Town05, four weathers and three repeats, 48 runs in total, and the agent-blocking timeout is cut from 300 s to 60 s [supp. Validation Protocol]. The paper does not state what the plus-minus values are.

Leaderboard tab. 1, main rows:

tab. 1. TCP-SB uses separate encoders for the two branches, and TCP-Ens averages TCP and TCP-SB §4.2. The authors state 13.291 points over the best multi-sensor entry, LAV §1.

Control-only versus trajectory-only, infractions per km with the share that happened while turning tab. 2:

Ablation, one component added at a time tab. 3. Rows before the last use only the control branch §4.4:

Multi-task versus ensemble tab. 4. Parameters, FLOPs and FPS are as printed. Hardware for the FPS figures is not stated:

The alpha sweep is only in a box plot: three trials per alpha. Alpha = 0 is poor, while plain averaging and the reversed mixing have comparable scores fig. 5. The authors show attention maps for two cases and state that the trajectory-guided attention focuses on different image regions at different future steps fig. 4 [supp. Qualitative Results]. The paper also reports that the control-only model without trajectory guidance pools its image feature once, so a single heat map cannot show per-step focus [supp. Qualitative Results].

Paper vs code

The paper links OpenPerceptionX/TCP. That repository's README says the project moved to OpenDriveLab/TCP. The card uses the moved repository at commit 73cc1ddfb615439b3f035c0285f5c9ffce350418. The supplement says the released code would be under CC BY-NC 4.0 [supp. License of Assets]. The repository's LICENSE file is Apache-2.0 LICENSE and the README says the same README.md. Those two statements conflict, and the card records the LICENSE file. No code is quoted here, only cited by path and line.

Discrepancies found, each with both citations:

• Pooled image vector. The paper average-pools the feature grid §3.2.1. The code instead uses the 1000-value output of the ResNet-34's final fully connected (ImageNet classifier) layer resnet.py model.py. Both the trajectory and control inputs use this vector model.py model.py.
• What feeds the temporal module. The paper feeds the current predicted action §3.2.2. The code feeds the Beta parameters of the current step model.py.
• Residual updates. The code updates both the waypoint sequence and the control feature with residual steps, adding an output to the previous value model.py model.py. The paper gives no such form.
• Waypoint decoder inputs. The code feeds the target point and the previous waypoint to the trajectory GRU at every step, starting from zero model.py model.py model.py. The paper does not describe per-step inputs.
• Waypoint loss. The paper sums L1 over the K steps eq. 4. The code averages over batch, K and both coordinates train.py. The scale differs.
• Loss weights. The supplement gives lambda traj = 1, lambda ctl = 1 and lambda aux = 0.05, then one sentence that assigns 0.001 to speed and value regression, which can be read two ways. The code uses speed weight 0.05 and value weight 0.001 config.py train.py. The paper's future-step feature terms in eq. 5 have no lambda F, but the code multiplies them by 0.05 train.py.
• Action loss. The code averages the KL over the two action dimensions with weight 0.5 each train.py train.py.
• Action decoding. The paper does not say how an action is read from a Beta distribution. The code uses the mode when both parameters exceed 1, edge values when one is at most 1, and the mean when both are at most 1, then rescales to [-1, 1] model.py. The sign of that value picks throttle or brake model.py.
• Initialisation from Roach. The code loads the value heads and the Beta heads from the Roach checkpoint at start-up, and the checkpoint is in the repository train.py config.py. The paper does not state this. Loading uses strict=False at inference tcp_agent.py.
• Expert label overrides. For samples where only the rule-based detector braked, the code sets the first action dimension's Beta parameters to fixed values, 0.8 and 5.5, instead of the expert's data.py. The paper does not mention this.
• Target point. Training computes a target point and then overwrites it with a variant that swaps the axes data.py. The agent computes it another way tcp_agent.py. The two frames are not checked against each other here.
• Training towns. The default config trains on Town01, 03, 04, 06 and validates on 02, 05, 07, 10 config.py. That matches the ablation split, not the leaderboard model trained on all towns §4.1.
• Dataset size. The text says 420K samples §4.1. The supplement table sums to 410,598 [supp. Dataset Statistics].
• Batch size. The paper says 128 [supp. Implementation Details]. The code default is 32 train.py.
• Augmentation. Training applies an image augmentation pipeline when img_aug is on, which is the default config.py data.py augment.py. The paper does not mention augmentation.
• Speed scaling. Speed is divided by 12 in training and in the agent train.py tcp_agent.py. The paper does not state this.
• Leaderboard configuration. The repository has one model class, one checkpoint, and fusion alpha 0.3 for both situations model.py tcp_agent.py tcp_agent.py. It has no TCP-SB or TCP-Ens model, no asymmetric alpha, and no max-brake rule. Those produced the leaderboard entries [supp. Implementation Details] §4.2. The leaderboard numbers are therefore not reproducible from this repository alone.
• Turning rule timing. The agent appends the fused steer to the window after computing the action, and it sets the situation for the next step tcp_agent.py. The paper does not state the lag.
• Brake and throttle post-processing. The agent zeroes the trajectory brake when it is below 0.05, or when the trajectory throttle exceeds it. It caps throttle at 0.75 after fusion, and zeroes throttle when brake exceeds 0.5 tcp_agent.py tcp_agent.py tcp_agent.py. The paper does not mention these.
• PID rules. The desired speed is twice the mean segment length, because waypoints are two per second model.py. The steer uses one aim point chosen from the waypoints, with a fallback to the target point model.py. The paper describes only the two PIDs.
• Validation loop. The validation future-action loop covers K - 1 steps, not K train.py. This affects the reported validation loss only.

Pitfalls

• Partial ablation rows. The ablation rows use the control branch alone until the final fused row §4.4. Only the last row is the full model.
• Spread versus gains. The plus-minus values are not defined in the paper. Some gains are of the same size as the spread: the attention row is 46.08 +- 3.47 against 42.87 +- 4.77 for the temporal row tab. 3. Check those gains before relying on them.
• Expert dependence. Labels come from Roach, including its value and latent features, so the student's targets are limited to what Roach does §3.1 roach_ap_agent.py.
• Simulator only. Closed-loop results come from one simulator and one leaderboard §4.1. The supplement warns that direct real-world deployment could cause accidents [supp. Broader Impact].
• Single-camera limits. The authors name failures from a red vehicle entering the path outside the front view, and from the agent misjudging other vehicles' paths [supp. Failure Cases].
• Heavy data and compute. The repository says the dataset is about 115 GB, plus a CARLA install README.md. TCP-Ens needs about three times the FLOPs of TCP, 25.61G against 8.54G tab. 4.
• Numerically sensitive steps. The Beta readout (mode versus mean) changes the action sign and magnitude model.py. The PID integral is a mean over a fixed-length window, so it depends on the control rate model.py.
• Turning threshold. The rule uses steer magnitude above 0.1 over 20 control steps, and "more than half" is the code's choice where the paper says "half" [supp. Implementation Details] tcp_agent.py.
• Licence. The repository LICENSE is Apache-2.0, but the paper's supplement says CC BY-NC 4.0 for the released code and training data. Resolve this before publishing any code snippet. This card does not quote code.

Lineage

• Builds on: the Roach expert §3.1 for supervision and value and latent features. The trajectory branch is inspired by Transfuser's GRU waypoint decoder §3.2.1. The data split and route protocol follow LAV §4.1 [supp. Validation Protocol].
• Related: FASNet predicts future positions as an auxiliary task with a kinematic loss §2.2. Ensemble methods are discussed in §2.2.
• Compared against: the entries in tab. 1, including LAV, Transfuser, Transfuser+, WoR, MaRLn and NEAT.
• Superseded by: not stated in this wiki.