Deep Reinforcement Learning framework for Autonomous Driving

In plain words

Teaching a car to drive is hard to write as a list of labelled examples, because other drivers, pedestrians and road works keep changing the situation. This paper asks whether a reinforcement learning agent can learn to drive instead: it tries actions, gets a reward signal for good driving, and improves from its mistakes.

The proposed system is one neural network pipeline that goes from raw sensor readings to driving actions. It combines sensors into one input, uses an attention mechanism to look only at the parts of that input that matter, keeps a memory over time so it can cope with things it cannot see right now, and learns the driving choices with a standard reinforcement learning method.

The authors test the idea on lane keeping in a racing simulator, not on a real car. The paper reports that the agent learns to keep its lane, but it gives no numerical measurements.

TL;DR

• Proposes an end-to-end deep reinforcement learning pipeline for autonomous driving §4.
• The pipeline fuses sensors, applies attention and convolution, integrates over time with an LSTM, and then plans with a DQN or DDAC head §4.1–§4.3.
• Tested only on lane keeping in TORCS, with a track-position input and discrete or continuous steering, throttle and brake outputs §5.
• Results are qualitative: "successful lane keeping" with no metrics, baselines or run counts §5. The attention and recurrence parts are not evaluated §4.1.2 §4.2.

Problem

Driving involves multi-agent interaction, and it is hard to build a supervised dataset that covers every scenario §2.1. Standard autonomous driving decouples the system into sub-problems such as detection and odometry, which may be harder than driving itself and may not combine well into a driving goal §2.1. Reinforcement learning models the driving goal directly through a reward, but real-car training is risky, so most work uses simulators such as TORCS or Unity §2.1.

Partial observability is the other issue. Occlusion during tracking and localisation means a single observation does not reveal the true state, so the agent has to integrate information over time §3.3 §4.2. Raw camera images are high dimensional, while the information needed for driving is much lower dimensional, so filtering the input could lower compute on embedded hardware §1. The paper states these as motivations; it does not measure compute.

Method

The paper builds on three existing methods. The first two are the basis of the planner, and the third and fourth feed the proposed pipeline.

• MDP and Q-learning. The problem is a Markov decision process with states, actions, discount factor, reward and transition probabilities §2.2. The value of a state is the expected discounted sum of rewards under a policy §2.2. Q-learning updates an action-value table with a temporal difference step built on the Bellman equation §2.3.
• DQN. When states are continuous, the Q-function is a parameterised neural network Q(s, a, w). Its weights are fitted by minimising the squared error between the Q-value and the one-step target §3.1. The paper says DQN works with discrete actions §3.1.
• DDAC (continuous actions). An actor π(s, u) outputs actions and a critic Q(s, a, w) evaluates them §3.2. The critic is trained as in DQN. The actor gradient is the critic's gradient with respect to the action, times the actor's gradient with respect to its weights §3.2. The paper names this family "deep deterministic actor critic" and cites policy gradient methods §3.2.
• Recurrence (DRQN, LSTM). An LSTM is added to the Q-network to handle partial observability §3.3 §4.2. The paper treats an RNN or LSTM as a learned Bayes-filter-like update of a hidden state from the previous state and the new input §4.2.
• Attention (DARQN, glimpse network). A glimpse network chooses parts of the input to attend to, and the paper proposes using a glimpse and action network in place of a plain attention filter over the CNN features §3.4 §4.1.2. The glimpse motion is learned with REINFORCE §3.4.

The proposed pipeline is trained end to end, and the same RL error is backpropagated through all layers §4.3. It has three stages:

1. Spatial aggregation §4.1. Sensor readings are grouped into one raw input vector, and the network learns how much each sensor matters §4.1.1. A glimpse and action network then selects which parts of this input the convolution sees §4.1.2 fig. 4.
2. Recurrent temporal aggregation §4.2. LSTM-style gates decide what to keep from the previous hidden state and what to add from the new input §4.2.
3. Planning §4.3. A Q-network head (discrete actions) or a DDAC actor-critic head (continuous actions) is trained with the same objective as in the DQN and DDAC sections §4.3.

# Q-learning target and squared TD error  [§2.3] [§3.1]
# Input:  Q-network q; batch of (state, action, reward, next_state)
# Output: scalar loss to minimise

def td_loss(q, batch, gamma):
    # Q(s, a; w) for the action that was taken  [§3.1]
    predicted = q(batch.state)[batch.action]             # (B,)

    # Best next-state value, max over actions  [§2.3] [§3.1]
    # The paper writes arg max here; see Pitfalls.
    best_next = max_over_actions(q(batch.next_state))    # (B,)

    # One-step target r + gamma * best next value  [§2.3]
    target = batch.reward + gamma * best_next            # (B,)

    # Mean squared error between target and prediction  [§3.1]
    return mean((target - predicted) ** 2)

# Proposed pipeline, one time step  [§4] [fig. 4]
# Input:  sensor readings and car state at this step
# Output: driving action (steer, gear, brake, acceleration)  [§5]

def drive_step(sensor_readings, hidden_state):
    # Fuse all sensor readings into one raw vector  [§4.1.1]
    fused = concatenate(sensor_readings)

    # Glimpse network picks the region the kernel attends to  [§4.1.2]
    glimpse = glimpse_network(fused, hidden_state)

    # Convolution only on the attended region  [§4.1.2] [§3.4]
    features = cnn(glimpse)

    # LSTM-style update over time  [§4.2]
    hidden_state = lstm(features, hidden_state)

    # Q-values for discrete actions, or actor output for continuous  [§4.3]
    action = planning_head(hidden_state)
    return action, hidden_state

Equations in the source are unnumbered (gather*), so they are cited by section. The only table is the state-function comparison in §4.2 §4.2.

Hyperparameters

Nothing numeric is stated for the training or network setup. The table lists every setting the paper names and marks the rest as not stated.

Results

Author-reported, qualitative only. The paper states no metric values and no baselines beyond the comparisons below. Setup: TORCS simulator with the Simulated Car Racing add-on; inputs are the track-position sensor and speed; outputs are steer, gear, brake and acceleration §5. The paper also names a multi-agent Unity scene fig. 2 and a training and driving video, but the video results are outside the paper text.

The paper reports no training curves, no lap or off-track statistics, and no results for the attention or LSTM components on their own §5.

Paper vs code

No official code.

Searches done for this card, all with no official result:

• The arXiv metadata and the paper text link only to demonstration videos §5 app.; no code link is given.
• GitHub repository search for torcs org:valeoai, deep reinforcement learning autonomous driving torcs valeo and reinforcement driving org:valeo returned no repositories.

No code is recorded, so there is nothing to compare the paper against.

Pitfalls

• Objective sign. The DQN section writes the objective as a maximum over the squared error, J(w) = max_w l(w) §3.1. The text then says the goal is to minimise the MSE §3.1. Implement it as a minimisation of the squared TD error.
• arg max in the target. The Q-learning update §2.3 and the DQN loss §3.1 write arg max over next actions where the target needs the maximum Q-value. The arg max returns the action, not its value. The pseudocode above uses max; this is an interpretation of the intent, not a stated detail.
• Discount term typo. The value expansion §2.2 repeats R(s₁) in the second and third terms. The standard form uses the next reward in each term.
• Target network and gradient flow are not stated. The paper does not say whether the target is held fixed during the update, whether a separate target network is used, or whether replay sampling is uniform. Pick these yourself and record that choice.
• Reward and termination are not stated. The lane-keeping reward, the crash condition and the episode length are all missing §2.1 §5. An implementer must design them, and results will depend heavily on that choice.
• Discretisation of actions is not stated. For DQN the actions are "tiled and discretized" §5, but the number of bins and the ranges are not given. The paper notes that tiling makes the steering abrupt §5.
• Input versus the raw-sensor claim. The introduction describes raw sensor input §1. The experiment uses a track-position signal and speed, not raw camera or LIDAR data §5. No pixel input was tested.
• Sensor fusion is not tested. The fusion design §4.1.1 is described, but the experiment uses only two signals §5.
• Attention and recurrence are not evaluated. The paper argues the glimpse network reduces computation and suits embedded hardware §4.1.2 §1, but no compute, memory or latency number is reported. It is also not stated whether the §5 experiments used the LSTM or the glimpse network.
• Glimpse training. The glimpse motion is learned with REINFORCE §3.4, which is a separate learning signal from the end-to-end RL error. The paper does not say how the two are combined in the proposed pipeline §4.3.
• Table notation is informal. The state-function table in §4.2 §4.2 lists RNN, Kalman, LSTM and GRNN rows. The GRNN row is the same as the LSTM row. The Bayes filter equation §4.2 has a mismatch in the integration variable. Treat the table as an informal analogy, not as derivations.
• Apprenticeship mixing. ε is described as a share of actions taken from the model, reduced gradually §5. The schedule, and whether mixing happens per step or per episode, is not stated.
• Qualitative claims. "Successful", "faster" and "better performance" §5 have no measurements behind them. Do not cite them as numerical evidence.

Lineage

• Builds on: DQN, arXiv:1312.5602 (Mnih et al., 2013), as cited in §3.1.
• Builds on: DRQN, arXiv:1507.06527 (Hausknecht and Stone, 2015), as cited in §3.3.
• Builds on: DARQN, arXiv:1512.01693 (Sorokin et al., 2015), as cited in §3.4. The paper proposes the glimpse and action network as a replacement for DARQN's attention filter §4.1.2.
• Compared against: no method is compared in the paper's experiments §5.
• Superseded by: not stated in this wiki.