CLIP-RLDrive: Human-Aligned Autonomous Driving via CLIP-Based Reward Shaping in Reinforcement Learning

In plain words

A self-driving car has to decide when to slow down, wait or speed up at a four-way intersection with no traffic lights. Reinforcement learning teaches such a decision by giving the learner a score (reward) for each action. Writing that score by hand is hard, because the car has to be rewarded for behaviour that is safe and sensible, not just for finishing.

This paper adds a second score from CLIP, a model that matches pictures with sentences. CLIP looks at the view from the car and says which of three actions fits the scene. When the car's action agrees with CLIP, it gets some extra reward, which is added to the simulator's usual score. The authors train two learning methods, DQN and PPO, with and without this extra score. In their own tests, the DQN version with CLIP does best. The PPO version with CLIP has a lower success rate than plain PPO.

TL;DR

• Trains a DQN or PPO agent to turn left at an unsignalized intersection in highway-env, with a grayscale four-frame view as input §2.3 tab. 4.
• Adds a CLIP-based term to the reward, using a ViT-B/32 image encoder that is partly re-trained on 500 scene and instruction pairs §2.4 app. A.2.
• Author-reported: CLIP-DQN gets 96% success and 4% collisions with one other vehicle, against 70% and 30% for DQN without CLIP tab. 2.
• Code is public (MIT) but the CLIP fine-tuning, its dataset and its weights are not in the repository. The shipped no-CLIP baseline still runs the CLIP reward (see Paper vs code).

Problem

Writing a reward for driving at unsignalized intersections is hard, and sparse or slow feedback slows learning §1.1. The paper frames the agent as a partially observable decision process §2. Other vehicles spawn at a set probability during the episode tab. 4. It also has to balance speed against collisions and time-outs §3.2.

Method

Environment. The simulator is highway-env intersection-v1, with a 30 s episode and a left-turn task from a fixed start to a target §2.3 §3.1 app. A.3. The observation is a grayscale top-down image of size (128, 64) with the four latest frames stacked §2.3 tab. 4. The simulator runs at 15 Hz §2.4 tab. 4. The agent chooses a meta-action at 1 Hz §2.3. The three actions are Drive Slower, Maintain Speed (IDLE) and Drive Faster §2.3.

Agents. DQN uses a CNN over the stacked frames fig. 2 §2.1. PPO uses the standard clipped objective §2.2 eq. 8. Both are trained with Stable-Baselines3 §2.4.

DQN objective. The Q-network is trained to match a one-step target that uses a periodically copied target network §2.1 eq. 4 eq. 5:

y = r + \gamma \max_{a'} Q_{\text{target}}(s', a'; \theta^-) \qquad [\text{eq. 5}]

\mathcal{L}(\theta) = \mathbb{E}\left[\left(y - Q(s, a; \theta)\right)^2\right] \qquad [\text{eq. 4}]

CLIP reward model. The image encoder is the ViT-B/32 model §2.4 app. A.2 tab. 3. Its upper layers are updated on a set of 500 road images, each paired with one short instruction, for 15 epochs with Adam §2.4 app. A.2 fig. 3. The text side encodes the three action sentences the agent can take §2.4. Cosine similarity between the image embedding and the text embedding of the expected behaviour gives the CLIP score eq. 12.

R_{CLIP} = \frac{\mathcal{E}_{image} \cdot \mathcal{E}_{text}}{\|\mathcal{E}_{image}\|\,\|\mathcal{E}_{text}\|} \qquad [\text{eq. 12}]

Combined reward. The simulator reward has a speed term, a collision term and a destination term, each with the values in §2.4 eq. 13. The paper then adds a weighted CLIP term to it eq. 14:

R_{Basic} = \begin{cases} r_{speed} & \text{efficient speed} \\ r_{collision} & \text{collision} \\ r_{destination} & \text{destination reached} \\ 0 & \text{otherwise} \end{cases} \qquad [\text{eq. 13}]

R_{Final} = R_{Basic} + W_c \times R_{CLIP} \qquad [\text{eq. 14}]

The paper's Algorithm 1 uses a different rule for the same step alg. 1. The CLIP term is added only when the agent's action matches the action CLIP suggests. Otherwise the basic reward is used alone. Eq. 14 and Algorithm 1 disagree; see Pitfalls.

# Per-step reward, Eq. 12 and 14 form  [eq. 12] [eq. 14]
# Input:  frame (grayscale stack), action, r_basic
# Output: reward used for the update

def shaped_reward(frame, action, r_basic):
    # 1. Embed the current frame with the adapted image encoder  [§2.4]
    image_embedding = clip_image_encoder(frame)                 # (512,)
    # 2. Embed the text that describes the expected behaviour  [§2.4]
    text_embedding = clip_text_encoder(expected_text(frame))    # (512,)
    # 3. Cosine similarity between the two embeddings  [eq. 12]
    r_clip = cosine_similarity(image_embedding, text_embedding)
    # 4. Add the weighted CLIP term to the basic reward  [eq. 14]
    return r_basic + W_c * r_clip                               # W_c = 1.2

# Gated variant described in Algorithm 1  [alg. 1]
# Input:  frame, action, r_basic, r_clip
# Output: reward used for the update

def gated_reward(frame, action, r_basic, r_clip):
    # 1. Ask CLIP which action the scene suggests  [alg. 1]
    suggested = clip_suggested_action(frame)
    # 2. Add the weighted CLIP term only when the actions agree  [alg. 1]
    if action == suggested:
        return r_basic + w * r_clip
    # 3. Otherwise keep the basic reward  [alg. 1]
    return r_basic

Evaluation. After training, each agent runs 100 test episodes and the authors count successes, collisions and time-outs §3.2 alg. 2. Each episode ends on a collision or on arrival, or when time runs out §3.2 alg. 2.

Hyperparameters

Values from the paper's tables and text, with the repository value next to them where the code gives one. "Not stated" means the paper does not give it.

Results

Author-reported. Setup: highway-env intersection-v1, 100 test episodes per agent §3.2. The paper does not state the number of training runs, seeds or confidence intervals. Each row sums to 100%, which fits 100 episodes tab. 2.

The "Vehicles" column is the number of on-road vehicles in the test scenario tab. 2. The paper does not say how those scenarios are generated.

Training curves show the CLIP-based runs gaining reward over time, while the runs without CLIP barely move §3.1 fig. 5. Episode length is used to read collisions (short episodes) and time-outs (long ones) §3.1 fig. 6. Confusion matrices compare CLIP's suggested action with the agent's action; the paper reports good agreement for CLIP-DQN and poor agreement for CLIP-PPO when slowing down §3.2 fig. 7. The speed profile for CLIP-PPO is described as fluctuating §3.2 fig. 8. Action counts for CLIP-DQN show FASTER chosen most often and IDLE least often §3.2 fig. 9.

Paper vs code

Repository: erfun77/CLIP-RLDrive at c65b83982dff083468a0b71cb3bbdc4f9b67ac67 (main branch, shallow clone, MIT). The README cites this arXiv id and both authors. The owner handle "erfun77" resembles the first author's name, and that is the basis for treating it as official. A human should confirm this before status: reviewed. Searches that found it: GitHub repository search for "CLIP-RLDrive" and "RLDrive CLIP". The LICENSE file names Edouard Leurent (2018), the upstream highway-env author, so the MIT grant also covers the upstream highway_env/ code, not only the authors' additions.

• Reward formula. Eq. 12 uses cosine similarity eq. 12. The code takes a softmax over raw dot products of unnormalised CLIP features, across three fixed prompts ("Slow down.", "Proceed.", "Speed up.") [code: highway_env/envs/intersection_env.py#L216-L220]. Only the largest probability is used, so the term lies in [0, 1].
• Bonus rules. Eq. 14 always adds the term, and Algorithm 1 adds it only on agreement. The code adds a bonus on agreement and a fixed penalty of -2 on disagreement for two of the labels [code: highway_env/envs/intersection_env.py#L225-L234]. It also adds 0.15 for SLOWER and multiplies by 1.5 for FASTER. None of these constants is in the paper.
• Weight. The paper's W_c = 1.2 does not appear in the code. The code applies its bonus without any weight intersection_env.py.
• Prompts. The paper's training instructions are a set of 500 sentences. The code uses only the three fixed prompts above. The dataset and the 500-sentence list are not in the repository.
• Where the bonus lives. The bonus is inside IntersectionEnv.step intersection_env.py, not in a training wrapper.
• Baseline. intersection-v1 maps to ContinuousIntersectionEnv, which inherits step from IntersectionEnv [code: highway_env/init.py#L51-L53] intersection_env.py. So every intersection-v1 run, including the no-CLIP script highway_env/envs/vanilla.py, receives the CLIP bonus. The repository has no CLIP-free path. Whether the paper's no-CLIP numbers came from another build is not determinable from the repo.
• Fine-tuning. The loader reads a weights file whose name ends in "V5" from an absolute Windows path intersection_env.py, and the top-level script reads "V6" intersection_GrayScale.py. Neither weights file, nor the fine-tuning script, is in the repository.
• Demo script. intersection_GrayScale.py runs a saved DQN model with the CLIP code commented out intersection_GrayScale.py. It is an evaluation loop, not the training recipe.
• Other scripts. scripts/sb3_highway_*.py use highway-fast-v0 with different settings, such as gamma 0.8 and 1e5 steps [code: scripts/sb3_highway_dqn_cnn.py#L34-L49]. These are not the paper's intersection setup.
• Not inspected. The repository ships model archives (*.zip) and a Best Models/ folder. They were not unpacked, so the trained configurations they contain are not checked here.
• Speed term. The paper's speed reward is a fixed value at an efficient speed eq. 13. The code gives a linear value from 0 to 1 over the speed range, times the speed weight, clipped to [0, 1] [code: highway_env/envs/intersection_env.py#L154-L164].
• Arrival. On arrival the code replaces the reward with the arrival reward intersection_env.py. The CLIP bonus is then added after the step, so it still applies on the arrival step [code: highway_env/envs/intersection_env.py#L225-L234].
• Off-road. The on-road factor multiplies the basic reward only [code: highway_env/envs/intersection_env.py#L145]. The CLIP bonus is not multiplied by it, so it still applies off the road.
• Action labels. The code labels the three actions SLOWER (0), IDLE (1) and FASTER (2) action.py, which matches §2.3.

Pitfalls

• Two reward rules in the paper. Eq. 14 always adds the CLIP term, while Algorithm 1 adds it only when the action agrees with CLIP. An implementation must pick one. The code uses a third rule (see Paper vs code).
• Cosine versus softmax. A cosine similarity lies in [-1, 1]. The code's term lies in [0, 1] and includes fixed constants (0.15, 1.5 and -2). Scaling the CLIP term by 1.2 does not reproduce the code's numbers.
• Stale frame across episodes. last_obs is a module-level global [code: highway_env/envs/intersection_env.py#L26]. It is set in step [code: highway_env/envs/intersection_env.py#L241] but never cleared on reset. The first step of every episode after the first therefore scores the last frame of the previous episode. The very first step of the process gets no bonus intersection_env.py.
• One frame, not the stack. CLIP sees index 3 of the four-frame stack [code: highway_env/envs/intersection_env.py#L206], rendered by matplotlib to a PNG and then resized to 224 pixels by the CLIP preprocessing. The paper does not describe this path, and the render changes pixel values.
• Pixel rendering. The frame is written to disk and reopened on every step intersection_env.py. The code also calls plt.show() on every step intersection_env.py. Both slow training and can block in an interactive backend.
• Unnormalised features. The code takes the dot product of raw image and text features, without normalising them [code: highway_env/envs/intersection_env.py#L216-L219]. The paper's cosine formula eq. 12 would normalise them. The softmax is therefore sensitive to feature norms, and the paper does not say which form was used.
• Paper's gamma values. The paper gives DQN gamma 0.95 and PPO gamma 0.99 tab. 1. The code's DQN uses 0.99 and its commented PPO uses 0.95. One of these may be a transcription error, and the paper does not say which.
• Fine-tuning ambiguity. The text says the upper layers are tuned §2.4, the appendix says "the last layer" app. A.2, and the weights file is named "upper layers". tab. 3 lists a learning rate of 5e-4 for this setup, but the repository has no training script to confirm it.
• Advantage formula. Eq. 9 ends with the term δ_{T-1} but raises γλ to the power T-t+1, which does not match the usual index for that term eq. 9. Check against the PPO implementation actually used.
• Eq. 1 and horizon. Eq. 1 sums to T, while the text speaks of an infinite horizon §2 eq. 1. The text also calls the model a partially observable process and, in the same sentence, a fully observable Markov decision process §2. Pick the finite-horizon reading for implementation.
• Test vehicles. The paper's training uses 5 initial vehicles tab. 4, while its test table uses 1, 3 and 6 tab. 2. The evaluation script uses 2 intersection_GrayScale.py. The three numbers do not match one another.
• Training length. The paper trains for 8000 steps §3.1. The code's training call is 6000 steps and is commented out [code: intersection_GrayScale.py#L80].
• Grayscale weights. The intersection scripts use RGB weights [0, 0, 0.05] intersection_GrayScale.py. The highway script uses standard luma-style weights sb3_highway_dqn_cnn.py. The paper does not state which is used.
• Training-time CLIP cost. The CLIP forward pass runs inside env.step, which the simulator calls once per meta-action, so at the 1 Hz policy rate abstract.py. The per-step PNG write is the larger cost.
• No variance reported. The table has no seeds, error bars or per-scenario counts, and the number of training runs is not stated. Differences of a few points should not be read as significant.
• Plain PPO beats CLIP-PPO on success. The text says CLIP-PPO beats plain PPO on training reward §3.1. The test table shows the opposite on success at every vehicle count: 38 against 78, 38 against 60 and 26 against 56 tab. 2.

Lineage

• Builds on: DQN and PPO as cited in §2 (keys mnih2013playing, schulman2017proximal); CLIP with a ViT-B/32 image encoder (keys radford2021learning, dosovitskiy2020image) §1.2 §2.4; highway-env and OpenAI Gym §1.2; Stable-Baselines3 §2.4. No cards in this wiki exist for these works yet.
• Compared against: plain DQN and PPO with the same environment tab. 2. The paper also cites related vision-language reward work, such as the CLIP zero-shot reward of Rocamonde et al. and LAMP §1.2. No cards exist for these.
• Superseded by: not known from this wiki.