DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving

In plain words

Self-driving cars usually have a decision module between the high-level goal ("take the next exit", "overtake") and the low-level steering and throttle. It picks coarse actions such as keep, accelerate, follow the lane or change lanes, and a motion planner turns those actions into real controls. Large language models can reason about a scene and explain themselves, but their answers are words, and words cannot steer a car directly.

DriveMLM bridges that gap. It is a multimodal language model that reads camera images, a LiDAR point cloud, the driving rules and the user's request. It answers with one choice from a small fixed list of speed and path actions, plus a sentence explaining why. Because those actions match the ones a standard planner already understands, the same model can sit in a conventional driving stack. The authors test it in a driving simulator, where it drives a long route with a higher score than the rule-based Apollo baseline.

TL;DR

• DriveMLM replaces the decision module of a modular driving stack with a multimodal LLM that outputs one speed decision and one path decision, plus an explanation §3.1 §3.2.
• Input is multi-view camera images over time, a LiDAR cloud, a system message with the traffic rules and decision definitions, and the user's text instruction §3.1.
• Trained on 280 hours of CARLA driving data the authors collected, with decision labels from rules and explanations written or refined by humans §4.1 §3.4.
• Author-reported: 76.1 driving score and 0.96 miles per intervention on CARLA Town05 Long, against 71.4 and 0.76 for Apollo tab. 4.
• Code and weights are not released, so the results cannot be re-run from this paper alone.

Problem

LLM driving systems that existed before this paper output language or high-level text. Those outputs cannot be turned into vehicle controls, so the systems were not evaluated in closed loop on realistic simulators §1 §2. End-to-end driving models learn only from driving data and lack broad world knowledge, which the authors say limits corner cases and reasoning §1. Rule-based stacks use a behavioral planning module whose discrete decision states can be converted into control by a motion planner, but they need hand-written rules for every situation §1 §2.

The paper's assumption is that aligning an LLM's output with those decision states closes the gap §1.

Method

Overview. The system has three parts: (1) behavioral planning states alignment, (2) the MLLM planner (a multi-modal tokenizer plus an MLLM decoder), and (3) an efficient data engine §3.1 fig. 3. At each step the model outputs a decision state token and an explanation. The decision state goes to a motion planning and control module that computes the trajectory §3.1.

Decision states §3.2. The decision space has two parts, chosen together at each step:

• speed: KEEP, ACCELERATE, DECELERATE, STOP;
• path: FOLLOW, LEFT_CHANGE, RIGHT_CHANGE, LEFT_BORROW, RIGHT_BORROW.

The system message maps natural-language descriptions to these states. The authors put this mapping in the system message fig. 4. The card's text does not restate the definitions; see Pitfalls.

Multi-modal tokenizer §3.1 §3.3:

• Images: the input is I \in \mathbb{R}^{T \times N_I \times H \times W \times 3}, with T time steps and N_I views. A CLIP vision encoder extracts features for each image. A projector maps them to the text width. A temporal cross-attention layer then uses the current frame as the query and the flattened earlier frames as keys and values. The result is N_I \times N_Q visual tokens per step.
• LiDAR: the cloud L \in \mathbb{R}^{K \times 4} goes through a sparse transformer encoder (SST). To align it with image features, the authors train it to match the frozen image-LiDAR CLIP image features with a cosine similarity loss. A linear projection then gives one LiDAR token. At training time the points are projected into each camera view, and points outside the image are dropped. At test time no points are dropped §3.3.
• Text: the system message M and the user instruction U are embedded with the LLM's own token embedding §3.3.

MLLM decoder §3.3. The decoder reads the image, LiDAR and text tokens. It is trained with cross-entropy and next-token prediction. The output has a decision part (one path decision and one speed decision in a predefined format) and an explanation part. The example dialogue in the system message covers a caption question, a navigation question and an explanation question fig. 4.

# DriveMLM decision step  [§3.1] [fig. 3]
# Input:  views (T, N_I, H, W, 3); lidar (K, 4); text (system, user)
# Output: path decision, speed decision, explanation text

def drive_step(views, lidar, system_msg, user_msg):
    # Encode every camera view with the CLIP vision encoder  [§3.3]
    view_features = vision_encoder(views)
    # Project to the LLM width and fuse time steps per view with
    # temporal cross-attention; the current frame is the query  [§3.3]
    image_tokens = temporal_fusion(project(view_features))   # (N_I, N_Q, D)
    # Encode the point cloud into one token  [§3.3]
    lidar_token = project_lidar(lidar_encoder(lidar))         # (1, D)
    # Embed the system message and user instruction as text  [§3.3]
    text_tokens = token_embedding(system_msg, user_msg)
    # Decode the decision text, then the explanation  [§3.3] [fig. 4]
    decoded_text = mllm_decode(image_tokens, lidar_token, text_tokens)
    path, speed, explanation = split_output(decoded_text)
    # The decision states go to motion planning, which computes the
    # trajectory and the vehicle controls  [§3.1]
    return path, speed, explanation

Training objectives §3.3 §4.2. The paper describes two losses. The LiDAR alignment term is a cosine similarity loss against frozen image features. The decoder term is next-token cross-entropy. How the two are weighted is not stated.

# Training losses  [§3.3]
# Input: batch of (image, lidar, decision and explanation text)

def training_step(batch):
    # Frozen image-LiDAR CLIP encoder gives the target feature  [§3.3]
    target = frozen_clip_image_encoder(batch.image)
    # Push the LiDAR encoder output toward the target  [§3.3]
    lidar_feature = lidar_encoder(batch.lidar)
    align_loss = cosine_similarity_loss(lidar_feature, target)
    # Next-token cross-entropy on the decision and explanation text  [§3.3]
    lm_loss = next_token_cross_entropy(mllm(batch), batch.text_targets)
    # The weighting between the two terms is not stated  [§3.3]
    return align_loss, lm_loss

Data engine §3.4. The authors build challenging scenarios in CARLA. Experts (human drivers or agents) drive them safely, and interaction data is recorded when an expert accepts a random driving request. Speed and path decision labels are generated from the expert trajectories with hand-written rules. Explanations are first generated from the nearby scene elements, then refined by human annotators. GPT-3.5 is used to widen their variety. Human annotators also refine the interaction annotations, which include both accepted and rejected requests §3.4.

Training and inference inputs §3.1 §4.2. Symbols for the inputs are given in §3.1: X_I \in \mathbb{R}^{N_I \times N_Q \times D}, X_L \in \mathbb{R}^{1 \times D}, X_M \in \mathbb{R}^{N_M \times D}, X_U \in \mathbb{R}^{N_U \times D}. N_M and N_U are the token counts of the system message and the user instruction. The paper gives no numeric values for them.

Equations. The paper has no numbered equations. The only formal notation is the tensor shapes in §3.1.

Hyperparameters

Results

Author-reported. Setup: CARLA Town05 Long closed loop, and a Town05 open-loop set of human-driven routes. The model is DriveMLM with the settings above. The paper does not say whether baseline numbers were re-run or copied from earlier papers §4.4.

Closed-loop driving tab. 4. DS = driving score, RC = route completion, IS = infraction score, MPI = miles per intervention.

The intro gives MPI as 0.955 §1. The table gives 0.96.

Open-loop evaluation on Town05 tab. 3. Each method is scored on the decisions it outputs and on the explanation text.

LLaVA 1.5 and InstructBLIP were not fine-tuned. They received a few input/decision examples §4.2. Apollo was replayed on the recorded scenarios §4.3.

Ablation tab. 5. MV = multi-view images, CT = concatenated temporal tokens, TQ = temporal QFormer, PC = point clouds.

The text gives different gap values for these rows §4.5; see Pitfalls. Use the table values.

Inference latency tab. 6. Seconds per token. The table has no row for DriveMLM itself.

Hardware is not stated §4.6.

Zero-shot on nuScenes. Decision accuracy 0.395 on 6,019 annotated validation frames. The authors describe this as qualitative evidence §4.6 fig. 7. The annotation is not published.

Headline claims. The abstract reports gains of 3.2 points over Autopilot and 4.7 points over Apollo on Town05 Long [abstract]. The 4.7-point gain over Apollo matches tab. 4. No Autopilot result appears in the paper's tables.

Paper vs code

No official code. The paper has no code link, and the arXiv metadata has no code link. A GitHub search for "DriveMLM" returned OpenGVLab/DriveMLM. Its README cites this paper with the same author list. Its "Release code and models" item is unchecked, the dataset item is unchecked, and the LICENSE link is dead. The repository holds a README and images, and no source code, no configs and no weights. Nothing can be compared line by line, so no code citation is used and no code field is set.

Pitfalls

• Autopilot or Apollo. The abstract and §3.1 align the output with the Autopilot behavioral module. The introduction says the decision states were taken from the Apollo behavioral module §1. Only Apollo is run in the closed-loop experiments §4.4. How the Autopilot integration works is not stated.
• Decision definitions are elided. The system message in fig. 4 shows "FOLLOW means ...", "KEEP means ...". The meaning of each state, and the rules that link words to states, are not written out. An implementation needs these definitions from the authors or the data engine.
• Decision-to-control interface. The paper says the decision goes to a motion planner that computes the trajectory §3.1. It does not describe the interface, the horizon, or the mapping from a borrow or change decision to a lane target.
• LiDAR encoder conflict. §3.3 says the SST encoder is randomly initialized. §4.2 says a GD-MAE encoder fine-tuned on ONCE is used. The paper does not say which one the reported model uses. In the ablation the LiDAR input does not help §4.5.
• Two different image encoders. The MLLM uses EVA-CLIP ViT-g/14 §4.2. The LiDAR alignment uses CLIP ViT-L/14 at training time only §3.3. The two are not interchangeable.
• Meaning of N_Q. §3.1 defines N_Q as the number of features in the last CLIP layer. §4.2 defines it as the number of query tokens in the querying transformer, 32. Use §4.2 for the value and treat the §3.1 definition as unverified.
• Temporal length. The default is T = 2 §4.5. The paper also describes images from time −T to 0 §3.3. It does not say how many frames are used, how they are sampled, or whether the default counts frames or seconds.
• Undefined pre-trained model. §4.2 says the MLLM is built from a pre-trained "husky" model. The paper does not define it. Whether it is released is not stated. Reproduction is not possible without it.
• Text and table gaps. §4.3 gives LLaVA an accuracy of 40.97%. tab. 3 gives 22.92%. §4.5 gives an 18.19% accuracy gain from multi-view images, and tab. 5 gives 64.54 − 47.83 = 16.71 points. The 7.4% gain from temporal QFormer over concatenation is not what tab. 5 gives (75.23 − 67.22 = 8.01 points). The speed F1 gain of 0.05 matches TQ versus MV, not TQ versus CT. Use the table values.
• Driving score is not RC × IS in the table. The paper says DS is the product of RC and IS §4.2. Table values give 98.1 × 0.78 ≈ 76.5 for DriveMLM and 92.2 × 0.80 ≈ 73.8 for Apollo, against 76.1 and 71.4 reported. DS is presumably computed per route before averaging. The paper does not say so.
• Fair comparison. The fine-tuned model is compared with few-shot LLaVA and InstructBLIP §4.2. The Apollo open-loop numbers come from a replay §4.3. The paper does not state whether the closed-loop baselines were re-run.
• Latency for DriveMLM is missing. The paper argues that real-time decisions matter §4.6, and reports latency only for other models tab. 6. The DriveMLM decoder size and the latency of its own inference are not reported.
• Data is not public. The datasets are "available from the corresponding author on reasonable request" (Declarations section). The annotation and the nuScenes labels are not released. Open-loop results cannot be reproduced.
• Label generation. The decision-rule set and the GPT-3.5 version and prompt are not stated §3.4. Explanation quality is judged only by BLEU-4, CIDEr and METEOR §4.3.
• Training pipeline. The paper says the QFormer and LLM are trained §4.2. Whether the LiDAR alignment and the decoder are trained in one stage or two, and whether the visual encoder is frozen, is not stated.

Lineage

• Builds on: LLaMA-7B as the language model and EVA-CLIP ViT-g/14 as the visual encoder §4.2; CLIP ViT-L/14 for image-LiDAR alignment §3.3; the SST sparse transformer §3.3; GD-MAE for LiDAR §4.2; the CARLA simulator §2.
• Compared against: LLaVA 1.5 and InstructBLIP in open loop §4.3; Apollo in both open and closed loop §4.3 §4.4; Roach, Interfuser and ThinkTwice in closed loop §4.4.
• Related LLM driving work: DriveGPT4, DriveLM, LanguageMPC, DiLu and others that use language for driving decisions, but are not evaluated in closed loop §1 §2.
• Superseded by: not stated in the paper.