Double Q-learning

Overview

Double Q-learning changes the bootstrapped target of Q-learning. The next action is chosen by one network and its value is read from another. In the Double DQN form, the evaluation network selects the action and the target network scores it 2412.02574 §4.1. The card does not state why the authors split the two roles. Only one card in the library uses the variant, so this page describes one paper's setup and does not compare variants.

Variants

Double DQN with prioritized replay

The agent picks changes to a driving simulator's scene. Its target is computed as described above, with the selection and evaluation networks sharing one architecture 2412.02574 §4.1. The Q-network has four layers: an input, two hidden ReLU layers and a linear output with one value per action. Hidden sizes are not stated 2412.02574 §4.1. The action set has 45 discrete actions 2412.02574 §3.2, and the state has 19 values 2412.02574 §3.1.

Transitions are stored in a replay buffer with prioritized sampling. The paper ties a priority to how well the selection network evaluated an action. It gives no values for the priority exponents 2412.02574 §4.1. Exploration starts at ε = 1.0 and falls to 0.1, using a decay schedule the paper does not spell out 2412.02574 §4.1. The target network is copied from the evaluation network periodically, and the paper gives no period 2412.02574 §4.1. The loss function is not stated in the paper text 2412.02574 §4.1. The released code uses different values for several of these settings; see the card's Paper vs code section before implementing.

Card: Generating Critical Scenarios for Testing Automated Driving Systems.

When to use which

Implementation sketch

Pseudocode of the target step only. The paper gives no value for the discount factor γ, the learning rate or the target copy period.

# Double DQN target  [arxiv:2412.02574 §4.1]
# eval_net and target_net share one architecture  [arxiv:2412.02574 §4.1]

def double_dqn_target(reward, next_state, gamma, eval_net, target_net):
    # Selection: the evaluation network picks the next action  [§4.1]
    best_next = argmax(eval_net(next_state))
    # Evaluation: the target network scores that action  [§4.1]
    q_next = target_net(next_state)[best_next]
    # gamma is not stated in the paper  [§3]
    return reward + gamma * q_next


# Copy weights from eval_net to target_net every so often  [§4.1]
# The period is not stated in the paper.

Contradictions

None known. Only one card uses the variant, so no two papers in the library report different results for it.

Papers

• Generating Critical Scenarios for Testing Automated Driving Systems: uses a Double DQN with prioritized replay to choose scene changes for a driving simulator, and reports author-measured collision counts.