Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer

In plain words

A self-driving car needs to know what is around it from several cameras and a laser scanner, and it needs to know what the traffic lights and stop signs say. InterFuser feeds all of these sensor readings into one shared network, so the model can compare what the front camera sees with what the side cameras and the laser scanner see. It learns this by imitating a rule-based expert that runs in a driving simulator and can see the simulator's internal state.

The network makes three kinds of output. The first is the path the car should follow. The second is a grid map of the cells around the car, showing where other road users are, which way they face and how fast they move. The third is a short list of traffic facts: the light colour, whether a stop sign is ahead, and whether the car is at a junction. The grid map and traffic facts are a readable record of what the model believes, which is the "interpretable" part of the name.

A separate safety layer reads the grid map and limits how fast the car may go, so that it stays clear of the objects the model sees. The authors report that removing this layer lowers the driving and infraction scores. All results come from a simulator and are the authors' own.

TL;DR

• InterFuser is an end-to-end driving model: a CNN per sensor, one transformer encoder over all sensor tokens, and one transformer decoder whose queries produce waypoints, a 20x20 object density map and traffic-rule outputs §3 §3.2.
• A safety controller uses the density map and traffic-rule outputs to choose a desired speed and to stop for red lights and stop signs §3.3 app. B.
• Author-reported: first place on the public CARLA leaderboard with driving score 76.18, route completion 88.23 and infraction score 0.84 (accessed June 2022) tab. 1. Also best on Town05 and CARLA 42-routes benchmarks app. F.
• Code is the opendilab/InterFuser repository (Apache-2.0). The released baseline and controller differ from the paper in several places; see Paper vs code and Pitfalls before reimplementing.

Problem

Driving policies that map sensors straight to controls are hard to verify. Two needs are named: understanding rare, multi-sensor situations, such as a pedestrian stepping out from behind a parked car or a car running a red light, and knowing why the model acted as it did §1. The authors argue that earlier fusion methods either match camera and LiDAR features through a locality assumption or concatenate them, and that they rarely model interactions across views and sensors §1. Their stated gap is a single transformer that fuses several views and LiDAR in one stage, without a separate CNN-transformer stage per modality §1 §2. The paper also asserts that an end-to-end policy has no built-in safety guard §1.

Method

Inputs and outputs §3.1

1. Four RGB inputs: left, front and right cameras, plus a focus crop from the centre of the front image to see distant traffic lights §3.1.
2. One LiDAR input, converted into a bird's-eye-view histogram image §3.1 app. C.
3. Waypoint output: a path of L = 10 waypoints for the ego vehicle §3.1.
4. Object density map M of size R x R x 7, where R = 20 cells of 1 m. Channels give object existence, a 2-D offset from the cell centre, the bounding-box size, the heading and the velocity. The ego vehicle is at the origin and the y-axis points forward §3.1.
5. Traffic-rule outputs: traffic-light state, stop-sign presence and junction presence §3.1.

Encoder §3.2 eq. 1

1. Each sensor passes through a CNN backbone, then a 1x1 convolution to d channels, then the feature map is flattened into tokens §3.2.
2. Each token gets a fixed 2-D sinusoidal position encoding and a learned sensor embedding (one per sensor). The sum is the token embedding §3.2 eq. 1.
3. All tokens from all sensors are concatenated and passed through a standard transformer encoder with K layers (multi-head self-attention, MLP and layer normalisation) §3.2.

Decoder and heads §3.2 §3.1

1. The decoder has three query groups: L waypoint queries, R² density-map queries and one traffic-rule query. Learned positional embeddings are added to all queries §3.2.
2. Waypoints: a single-layer GRU predicts each waypoint from the previous hidden state and the matching query. It predicts a displacement at each step and sums them. Its initial hidden state comes from a 64-dimensional linear embedding of the goal location §3.2.
3. Density map: each of the R² density queries goes through a 3-layer MLP with ReLU, reshaped to R x R x 7 §3.2.
4. Traffic rules: the traffic-rule query goes through one linear layer §3.2.

Loss §3.2 eq. 2 app. A

1. The total loss is a weighted sum of a waypoint term, a density-map term and a traffic term eq. 2.
2. Waypoint term: L1 distance to the expert's waypoints, summed over the L steps app. A.
3. Density-map term: a probability loss plus a meta-feature loss. The probability loss averages the error over positive and negative labels separately, so rare objects count fully. The meta loss is computed only on cells that hold an object app. A.
4. Traffic term: binary cross-entropy for each of the light, stop-sign and junction outputs app. A.

Safety controller §3.3 app. B

1. Steering: a PID controller aims at the desired heading, the average heading of the first two waypoints §3.3.
2. Object recovery: a cell counts as an object if its existence probability is above threshold 1 (0.9), or if it is above threshold 2 (0.5) and is a local maximum among its neighbours. The second rule finds uncertain objects without detecting one object many times §3.3 app. G.
3. Tracking and prediction: a tracker records the history of each object. Future positions are extrapolated with a moving average of past motion §3.3.
4. Safe distance: the controller computes s1 and s2, the distances the ego vehicle can drive at 0.5 s and 1 s on the predicted route before a collision. Objects are enlarged by a margin s̄ app. B.
5. Desired speed: a linear programme maximises the speed for the first step, subject to the distance limits, speed limits (v_max) and acceleration limits (a_max). The second-step speed is a free variable that keeps the first-step choice feasible further ahead app. B.
6. Traffic rules: the car performs an emergency stop if the light is not green or a stop sign is ahead §3.3.
7. Speed control: a PID controller follows the desired speed §3.3.

# InterFuser inference per frame  [§3.1] [§3.2]
# Views: front, left, right, focus (RGB); lidar (BEV histogram)
def interfuser(views, lidar, goal):
    # Encode each sensor into tokens with sensor and position embeddings  [§3.2]
    tokens = []
    for view in views + [lidar]:
        feature_map = backbone_for(view)      # (d, H, W)
        tokens.append(tokenize(feature_map) + sensor_emb + position_emb)
    # Joint transformer encoder over all sensor tokens  [§3.2]
    memory = transformer_encoder(concat(tokens))
    # Decoder queries: waypoints, 20x20 density cells, one traffic query  [§3.2]
    hidden = transformer_decoder(queries, memory)
    # Waypoints: one GRU, seeded by the goal embedding  [§3.2]
    waypoints = gru_waypoints(hidden[waypoint_queries], goal)
    # Density map: per-cell MLP to 7 channels  [§3.2]
    density_map = mlp(hidden[density_queries])    # (20, 20, 7)
    # Traffic light, stop sign and junction logits  [§3.2]
    traffic = linear(hidden[traffic_query])
    return waypoints, density_map, traffic

# Desired speed from the safety controller  [§3.3] [app. B]
# s1, s2: safe distances at 0.5 s and 1 s, with objects enlarged by s_bar
# T: time step; v_max and a_max: speed and acceleration limits
def desired_speed(v0, s1, s2, v_max, a_max, T):
    # Maximise v1 subject to the distance, speed and acceleration limits
    # v2 is free and only constrains v1 through the second-step limit  [app. B]
    v1 = linear_program_max_v1(v0, s1, s2, v_max, a_max, T)
    return v1

Hyperparameters

Values as stated in the paper, then what the released code uses. Blank code cells mean not checked.

Results

Author-reported. The simulator is CARLA 0.9.10.1 with 8 towns and 21 weather settings. Training data came from a rule-based expert with privileged access to the simulator, sampled at 2 FPS: 3M frames, 410 hours §4.1. Metrics: driving score (route completion times infraction score), route completion (RC) and infraction score (IS) §4.1.

Public CARLA leaderboard, accessed June 2022 tab. 1:

Town05 Long, with adversarial events, ablating sensor inputs tab. 2:

Town05 Long, ablating architecture and controller tab. 3:

Other author-reported numbers:

• Town05 Short driving score 94.95 and Town05 Long 68.31, against WOR (64.79 and 44.80) and Roach app. F.
• CARLA 42-routes benchmark: driving score 91.84, route completion 97.12, infraction score 0.95, against NEAT 65.17 and WOR 67.64 app. F.
• Failure analysis of 45% of failing cases from missed detections, 15% from inaccurate speed or heading, and 15% from traffic-light misreads; the rest is not stated app. E.
• Relative effects stated in the text: removing the safety controller lowers driving and infraction scores by 15% and 22%; concatenating views instead of transformer fusion lowers them by 58% and 63% §4.3.

Paper vs code

Code: https://github.com/opendilab/InterFuser at code_commit f0be8ea (Apache-2.0). The paper names this repository in its abstract. The repository's README gives the training command used in interfuser/scripts/train.sh.

• Loss weights. The paper's table lists light 0.2, stop sign 0.01, junction 0.1, waypoints 0.4, density map 0.4, traffic 1.0 app. G. The training loop combines density 0.5, waypoints 0.2, speed 0.05, junction 0.05, light 0.1 and stop sign 0.01 train.py. The paper's table lists λ_pt and λ_map, which the training loop does not use under those names. The code's speed term has no counterpart in the paper's loss.
• Waypoint loss. The paper sums L1 errors over the L steps app. A. The code applies ten decreasing per-step weights (about 0.14 down to 0.04) and then averages train.py.
• Traffic classification. The paper uses binary cross-entropy app. A. The code uses nn.CrossEntropyLoss on two logits for each traffic head train.py interfuser.py.
• Density-map labels. The paper's probability loss uses indicator functions for labels equal to 0 or 1 app. A. The code writes a soft existence value per cell, (0.5 / max(0.5, distance))^0.5, from the nearest actor det_utils.py. Offsets are divided by 3.5 and box sizes by 3.5 and 2 det_utils.py. The training loss treats any label of at least 0.01 as positive train.py.
• Density-map meta channels. The paper lists six meta channels, including velocity x and velocity y app. A. The code stores one scalar speed in channel 6 and trains channels 1 to 5 as meta features det_utils.py train.py.
• Density-map head. The paper describes a 3-layer MLP with ReLU §3.2. The code's density head is a 2-layer MLP (Linear, ReLU, Linear, Sigmoid), and it receives the ego speed as 32 extra input values per cell interfuser.py interfuser.py.
• Waypoint head. The paper's waypoint head is a GRU §3.2. The class default is a transposed-convolution "heatmap" head. The released interfuser_baseline switches to gru interfuser.py interfuser.py.
• Goal input. The paper seeds the GRU with the GPS coordinates of the goal §3.2. The code embeds a target point expressed in the ego frame, computed from the route planner interfuser_agent.py carla_dataset.py.
• Per-sensor global tokens. The code adds a mean-pooled token for each sensor to the encoder input, besides the spatial tokens. The paper does not mention these interfuser.py interfuser.py.
• Shared RGB backbone. The code runs the same RGB backbone and projection on the front, left, right and focus views interfuser.py. The paper does not say whether the views share weights.
• LiDAR input. The paper's main text says a 3-bin histogram gives a two-channel image §3.1. Appendix C says three channels app. C. The code produces three channels: below-ground, above-ground and total counts utils.py.
• LiDAR grid. The paper says 14 m to each side app. C. The code's x-axis runs from -28 m to 29 m, with 224 bins, so the cell width there is about 0.25 m rather than 0.125 m. The y-axis gives 0.125 m cells over 28 m carla_dataset.py utils.py. Counts are clipped at 5 per cell and divided by 5 carla_dataset.py.
• Side-view scale. The paper scales the short side of side views to 160 px app. C. The leaderboard agent uses 146 px interfuser_agent.py.
• Object threshold. The paper's second threshold is 0.5 app. G. The code's peak rule uses 0.4 render.py.
• Pedestrian and bicycle size. The paper gives a scale factor of 2 app. G. The code's rendering enlarges boxes narrower than 1.5 m by 1.5 and marks this as a FIXME heatmap_utils.py.
• Desired speed. The paper solves a linear programme over 1 s with 0.5 s steps app. B. The released controller does not solve one. It renders the surrounding objects and tests self-car footprints at 0, 0.5, 0.75, 1, 1.5 and 2 s interfuser_controller.py interfuser_controller.py. It then applies the fixed formula min(4 d_0.5 - v - max(0, v - 2.5), v_max, 2 d_1 - 0.5 v - max(0, v - 2.5)) interfuser_controller.py.
• Speed limit. The paper's v_max is 6.5 app. B. The code's config sets max_speed = 5 interfuser_config.py.
• Safety margin. The paper's s̄ is 2 at 100% safety factor app. B. The code uses a two-part buffer of 2.5 and 1.2, added to a hero footprint of 2.45 x 1.0 m interfuser_config.py interfuser_controller.py.
• Controller rules not in the paper. Red light: after more than 1000 steps with a red-light probability above 0.7, the code ignores red lights for 80 steps interfuser_controller.py. Stop sign: the code holds the car for a set distance after a stop-sign trigger interfuser_controller.py. Stall: after 1200 stopped steps, it forces 12 steps of throttle 0.8 interfuser_controller.py.
• Tracker smoothing. The paper describes a moving average of object motion §3.3. The agent's moving-average coefficient is 0 in the config, so each new prediction replaces the last one interfuser_config.py interfuser_agent.py.
• Training hyperparameters. The paper's learning rates (2.5e-4 and 1e-4) and batch size (256) are consistent with each other only at batch 256 app. C app. G. The released script trains with 16 per GPU on 8 GPUs, a total of 128, which gives roughly 1.25e-4 and 5e-5 under the code's scaling rule train.py train.sh.
• Epochs and weight decay. The paper states 35 epochs app. G and a weight decay of 0.07 in the text app. C. The released script uses 25 epochs and 0.05 train.sh.

Pitfalls

• The released controller is not the paper's controller. Anyone who wants the paper's safety behaviour must write the linear programme from app. B themselves. The released heuristic rules (red-light block, stop-sign hold and stall recovery) change behaviour and are not described in the paper.
• Output labels and loss terms. The density labels are soft, not 0/1, and the speed channel is a scalar. A reimplementation that follows the paper's indicator loss and velocity-x/y channels will train on different targets than the released code.
• Unit typos in the table. The table gives a_max as "1.0 m/s" and v_max as "6.5 m/s²" app. G. Treat them as m/s² and m/s, respectively.
• Baseline versus default. The class default waypoints_pred_head is heatmap but the released baseline uses gru. Build the model from interfuser_baseline or the paper will not match the code.
• Scale of the waypoint loss. The released code averages over the 10 steps and 2 coordinates and applies step weights, while the paper sums. Loss magnitudes and any learning-rate tuning therefore do not transfer between the two.
• Grid geometry. The released LiDAR cells are not square. The x-axis cell width is about twice the y-axis width. Check this against the paper's 0.125 m cells before reusing the code's grid.
• Simulator only. All evaluations are in CARLA. The authors note that all of the work is in simulation and that generalisation to real roads is future work §5.
• Leaderboard numbers are dated. The leaderboard table was accessed in June 2022 tab. 1. Later leaderboard entries are not covered.
• Spread of results. The ablation rows report ± values of several points, and the paper does not say how many runs each row averages. Small differences in driving score sit inside those ranges tab. 2 tab. 3.

Lineage

• Builds on: TransFuser, a CNN-transformer fusion of camera and LiDAR for CARLA driving, which the paper calls a multi-stage design §1 §4.2. Also LBC and IA, which concatenate views §4.3.
• Peak rule: the local-maximum criterion for object detection follows two papers by Zhou et al., cited in §3.3.
• Compared against: TCP, LAV, TransFuser, Latent TransFuser, GRIAD, Rails, IARL and NEAT on the leaderboard tab. 1; CILRS, LBC, TransFuser, NEAT, Roach and WOR on Town05 app. F.
• Superseded by: not stated in this card's sources.