---
kind: concept
slug: double-q-learning
title: Double Q-learning
summary: >-
  Choosing the next action with one network and scoring that action with a second, target network, inside a bootstrapped Q-learning target. One card uses it: a Double DQN with prioritized replay that generates critical driving scenes (author-reported results). The card gives no comparison with plain DQN.
tags: [reinforcement-learning, q-learning, double-dqn]
papers: ["2412.02574"]
related: [deep-q-learning, experience-replay]
status: draft
updated_at: 2026-10-25
---

## Overview

Double Q-learning changes the bootstrapped target of Q-learning. The next
action is chosen by one network and its value is read from another. In the
Double DQN form, the evaluation network selects the action and the target
network scores it [arxiv:2412.02574 §4.1]. The card does not state why the
authors split the two roles. Only one card in the library uses the variant,
so this page describes one paper's setup and does not compare variants.

## Variants

### Double DQN with prioritized replay

The agent picks changes to a driving simulator's scene. Its target is
computed as described above, with the selection and evaluation networks
sharing one architecture [arxiv:2412.02574 §4.1]. The Q-network has four
layers: an input, two hidden ReLU layers and a linear output with one value
per action. Hidden sizes are not stated [arxiv:2412.02574 §4.1]. The action
set has 45 discrete actions [arxiv:2412.02574 §3.2], and the state has 19
values [arxiv:2412.02574 §3.1].

Transitions are stored in a replay buffer with prioritized sampling. The
paper ties a priority to how well the selection network evaluated an action.
It gives no values for the priority exponents [arxiv:2412.02574 §4.1].
Exploration starts at ε = 1.0 and falls to 0.1, using a decay schedule the
paper does not spell out [arxiv:2412.02574 §4.1]. The target network is
copied from the evaluation network periodically, and the paper gives no
period [arxiv:2412.02574 §4.1]. The loss function is not stated in the paper
text [arxiv:2412.02574 §4.1]. The released code uses different values for
several of these settings; see the card's Paper vs code section before
implementing.

Card: [Generating Critical Scenarios for Testing Automated Driving Systems](/papers/2412.02574).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Discrete actions, a neural Q-function, and transitions you can replay | Double DQN target with a periodic target copy and prioritized replay | This is the setup of the one card that uses the variant [arxiv:2412.02574 §3.2] [arxiv:2412.02574 §4.1] |
| Continuous actions | Not covered here | The card's agent chooses among a fixed set of 45 actions [arxiv:2412.02574 §3.2] |
| Need evidence that the split helps over a plain DQN target | Not available from this card | The reported comparisons are against DeepCollision and random search, not an ablation of the target [arxiv:2412.02574 §5.2.2] [arxiv:2412.02574 §6.1] |
| Compute budget or run time | Plan for long runs | The paper gives no hardware or runtime figures, and each action takes a 6 s simulated run [arxiv:2412.02574 §5.2.2] |

## Implementation sketch

Pseudocode of the target step only. The paper gives no value for the
discount factor γ, the learning rate or the target copy period.

```pseudocode
# Double DQN target  [arxiv:2412.02574 §4.1]
# eval_net and target_net share one architecture  [arxiv:2412.02574 §4.1]

def double_dqn_target(reward, next_state, gamma, eval_net, target_net):
    # Selection: the evaluation network picks the next action  [§4.1]
    best_next = argmax(eval_net(next_state))
    # Evaluation: the target network scores that action  [§4.1]
    q_next = target_net(next_state)[best_next]
    # gamma is not stated in the paper  [§3]
    return reward + gamma * q_next


# Copy weights from eval_net to target_net every so often  [§4.1]
# The period is not stated in the paper.
```

## Contradictions

None known. Only one card uses the variant, so no two papers in the library
report different results for it.

## Papers

- [Generating Critical Scenarios for Testing Automated Driving Systems](/papers/2412.02574): uses a Double DQN with prioritized replay to choose scene changes for a driving simulator, and reports author-measured collision counts.