---
kind: concept
slug: deep-q-learning
title: Deep Q-learning
summary: >-
  Learning action values with a parametric function instead of a table, usually with experience replay and a target copy. Six cards cover quantum, driving-test, driving-reward, robot and hardware-survey variants. Results are author-reported and do not compare variants on one task.
tags: [reinforcement-learning, quantum-machine-learning, variational-circuit, double-dqn, reward-shaping, fpga]
papers: ["1907.00397", "2411.06128", "2412.02574", "2412.10096", "2412.16201", "2610.28699"]
related: [experience-replay, actor-critic, policy-gradient, value-iteration, proximal-policy-optimization, reward-design, variational-quantum-circuit, double-q-learning]
status: draft
updated_at: 2026-11-01
---

## Overview

Q-learning keeps one value per state-action pair. That table grows with the
state space, so deep Q-learning replaces it with a function that has
trainable parameters [arxiv:1907.00397 §2.3]. The authors note that this
kind of training is unstable, because consecutive samples are correlated and
the regression target moves as the parameters change [arxiv:1907.00397 §2.3].

Two fixes recur: experience replay, which trains on random stored
transitions, and a target copy of the parameters, which supplies the
bootstrapped value [arxiv:1907.00397 §2.3] [arxiv:1907.00397 §4]. Most cards
use both, but not all. [arxiv:2412.10096 §5] keeps a buffer per machine
state and does not state a target network. [arxiv:2411.06128 §VII-B] names
DQN only as a baseline and gives no details. The hardware survey describes
both parts as well [arxiv:2610.28699 §3.4].

The update is the squared temporal-difference error between the current
value of the action taken and the reward plus the best discounted value
under the target copy, with a terminal state using the reward alone
[arxiv:1907.00397 eq. 3] [arxiv:1907.00397 alg. 1]. Variants change the
target. Double DQN lets the evaluation network choose the next action and
the target network score it [arxiv:2412.02574 §4.1]. Other cards take the
maximum over the target copy [arxiv:2412.16201 eq. 5]. The loss also
differs: squared error in [arxiv:1907.00397 eq. 3] and [arxiv:2412.16201 eq. 4],
Huber in [arxiv:2412.10096 eq. 1]. The discount factor is not stated in the
1907 paper [arxiv:1907.00397 §2].

## Variants

### Variational quantum circuit as the Q-function

The state index is written in binary and each bit sets one qubit with a
π-rotation. Two layers follow, each with a CNOT chain and one general
single-qubit rotation with three trained angles per qubit. The read-out is
the Pauli-Z expectation of each qubit plus a trainable bias, and the action
is the largest entry [arxiv:1907.00397 §4.1] [arxiv:1907.00397 fig. 5]
[arxiv:1907.00397 §6.1]. Frozen lake needs 24 rotation angles and 4 bias
values, 28 in total [arxiv:1907.00397 §4.1] [arxiv:1907.00397 tab. 2].
The paper's classical comparison uses a neural network from the ns3-gym work,
cited with a parameter count of 2n² + 2n [arxiv:1907.00397 §6.6]. That count
is the authors' citation and was not checked.
Card: [Variational Quantum Circuits for Deep Reinforcement Learning](/papers/1907.00397).

### Double DQN with prioritized replay

The agent chooses scene changes for a driving simulator. The evaluation
network picks the next action and the target network scores it [arxiv:2412.02574 §4.1].
Transitions are drawn in proportion to stored priorities, which the paper
ties to how well the selection network scored each action. It gives no
values for the priority exponents [arxiv:2412.02574 §4.1]. The Q-network
has two hidden ReLU layers and one output per action; hidden sizes are not
stated [arxiv:2412.02574 §4.1]. Exploration starts at ε = 1.0 and falls to
0.1 [arxiv:2412.02574 §4.1]. Card: [Generating Critical Scenarios for Testing Automated Driving Systems](/papers/2412.02574).

### Convolutional Q-network with a reward term

The Q-network is a CNN over four stacked frames [arxiv:2412.16201 fig. 2]
[arxiv:2412.16201 §2.1]. Its one-step target uses a periodically copied
target network [arxiv:2412.16201 §2.1] [arxiv:2412.16201 eq. 5]. The paper
adds a reward term from CLIP, weighted by 1.2 [arxiv:2412.16201 §2.4]. The
agent drives an unsignalized left turn. Author-reported, the DQN with that
term reaches 96% success with one other vehicle, against 70% for DQN without
it [arxiv:2412.16201 tab. 2]. The same table has plain PPO at 78%
[arxiv:2412.16201 tab. 2]. Card: [CLIP-RLDrive: Human-Aligned Autonomous Driving via CLIP-Based Reward Shaping in Reinforcement Learning](/papers/2412.16201).

### One Q-function per reward-machine state

Each state of the inferred reward machine has its own Q-function and its own
experience buffer. The buffers also store demonstrations, and all Q-functions
are updated from sampled batches at the end of each episode [arxiv:2412.10096 §5].
The paper uses the DQRM approach for this [arxiv:2412.10096 §5]. The loss is
the Huber TD loss [arxiv:2412.10096 eq. 1] [arxiv:2412.10096 eq. 2]. The
Q-values are pixel-wise maps over pick and place locations [arxiv:2412.10096 §5]
[arxiv:2412.10096 fig. 2]. Results are shown as learning curves only
[arxiv:2412.10096 fig. 5]. Card: [Reward Machine Inference for Robotic Manipulation](/papers/2412.10096).

### Baseline only

The warehouse paper reports a DQN accuracy ratio of 89.76 against 96.42 for
its own method [arxiv:2411.06128 tab. II]. The DQN architecture, training and
hyperparameters are not stated [arxiv:2411.06128 §VII-B], so the figure says
little about DQN as a method. Card: [Research on reinforcement learning based warehouse robot navigation algorithm in complex warehouse layout](/papers/2411.06128).

### Hardware accelerator for DQN

The survey treats DQN as a workload that FPGA or integrated-circuit hardware
runs. It writes the same squared loss over replay minibatches, with a target
network [arxiv:2610.28699 eq. 3] [arxiv:2610.28699 §3.4]. In the hardware
mapping it describes, forward passes run in pipelined matrix-vector units. The
replay buffer sits in on-chip block RAM or in DDR memory, and the target
network is a second module updated at a slower rate [arxiv:2610.28699 §3.4]
[arxiv:2610.28699 fig. 4]. Some designs split training into stages
[arxiv:2610.28699 §3.4]. The survey gives no target-update period. It notes
one design that uses minibatches of about 1 to 8 [arxiv:2610.28699 §3.4].

The survey's table of DQN designs lists reported results. Examples are 19.05
times faster than a GPU for one Virtex-7 design, and 671 times faster at
inference for an Ultra96-V2 design whose baseline the survey does not state
[arxiv:2610.28699 tab. 4]. The survey says baselines differ between studies,
so these values cannot be ranked against each other [arxiv:2610.28699 §3.9].
Some table rows look inconsistent with the survey's own column definitions, so
check a cited work before quoting one [arxiv:2610.28699 tab. 4]. The designs
it cites have no cards here, and nothing from them is used.
Card: [Reinforcement learning for hardware acceleration and system-level optimization on FPGA and integrated-circuit platforms: A survey](/papers/2610.28699).

## Settings in the cards

Values as each card records them. A row shows one paper's choices, not a
tested effect: no card compares these variants on the same task.

| Card | Q-function | Target | Batch | Learning rate | Exploration | Discount γ |
|---|---|---|---|---|---|---|
| [arxiv:1907.00397 §4] | 4-qubit circuit | copy every 20 steps [arxiv:1907.00397 §4] | 5 [arxiv:1907.00397 §4] | 0.01, RMSprop [arxiv:1907.00397 §6.2] | decay per episode or per step [arxiv:1907.00397 eq. 16] [arxiv:1907.00397 eq. 17] | not stated [arxiv:1907.00397 §2] |
| [arxiv:2412.02574 §4.1] | four layers, two ReLU hidden | Double DQN, period not stated [arxiv:2412.02574 §4.1] | not stated | not stated | 1.0 to 0.1 [arxiv:2412.02574 §4.1] | not stated [arxiv:2412.02574 §3] |
| [arxiv:2412.16201 tab. 1] | CNN over 4 frames | periodic, period not stated [arxiv:2412.16201 §2.1] | 32 [arxiv:2412.16201 tab. 1] | 0.0005 [arxiv:2412.16201 tab. 1] | linear decay, final ε not stated [arxiv:2412.16201 tab. 1] | 0.95 [arxiv:2412.16201 tab. 1] |
| [arxiv:2412.10096 §6] | one per machine state | not stated [arxiv:2412.10096 §5] | 16 [arxiv:2412.10096 §6] | 0.0001 [arxiv:2412.10096 §6] | 0.7 to 0.1, exponential [arxiv:2412.10096 §6] | not stated [arxiv:2412.10096 §3.1] |
| [arxiv:2411.06128 §VII-B] | not stated | not stated | not stated | not stated | not stated | not stated |
| [arxiv:2610.28699 §3.4] (survey, no settings of its own) | DQN on FPGA or IC, as described | "periodically updated", no period [arxiv:2610.28699 §3.4] | 1 to 8 in one design [arxiv:2610.28699 §3.4] | not stated | not stated | not stated [arxiv:2610.28699 eq. 3] |

## When to use which

The cards support the rows below. Each row says what its card shows.

| Situation | Use | Why |
|---|---|---|
| Small discrete task where a few dozen parameters are the goal | Variational circuit read-out, if you can simulate it | Frozen lake uses 28 parameters against 64 Q-table entries [arxiv:1907.00397 §6.6]. The reward result is simulator-only [arxiv:1907.00397 §6.4]. |
| Need real quantum hardware | Not supported by this card | Hardware runs are inference only, 5 radio episodes and 7 frozen-lake episodes, with no variance reported [arxiv:1907.00397 §6.5] [arxiv:1907.00397 tab. 3] [arxiv:1907.00397 tab. 4]. |
| More than 5 qubits | Not supported by this card | The authors say scaling beyond n ≤ 5 is untested [arxiv:1907.00397 §6.6]. Simulation cost grows with qubit count [arxiv:1907.00397 §4]. |
| Noisy device expected | Test first; one 2019 device snapshot was studied | The noisy 2- and 4-channel radio runs peak at about 110 iterations [arxiv:1907.00397 fig. 10] [arxiv:1907.00397 §6.4]. The noise study uses one snapshot [arxiv:1907.00397 app. Noise Information]. |
| Pixel input, discrete actions, and a reward that is hard to write by hand | DQN with a reward term, only if you can run the same reward model | DQN with the CLIP term beat DQN without it, 96% against 70% with one other vehicle [arxiv:2412.16201 tab. 2]. Plain PPO beat plain DQN in that table, 78% against 70% [arxiv:2412.16201 tab. 2]. |
| Search over many discrete scene changes in a simulator | Double DQN with prioritized replay | The paper compares against DeepCollision and random search, not against plain DQN, so the gain from Double DQN or prioritized replay is not isolated [arxiv:2412.02574 tab. 5]. |
| Task with a known or inferred sub-goal structure | One Q-function per machine state | Learning curves are close to a ground-truth machine agent, except on Stack-3 [arxiv:2412.10096 fig. 5]. Prototype-matching errors can stall the agent [arxiv:2412.10096 §6]. |
| Dedicated FPGA or integrated-circuit hardware | Hardware mapping above, with the survey as an overview | The survey gives the module split but no target-update period or training settings [arxiv:2610.28699 §3.4]. Its reported speedups use different baselines and do not rank designs [arxiv:2610.28699 §3.9] [arxiv:2610.28699 tab. 4]. |
| Large state space or long training | Not covered | No experiment in the 1907 card tests it [arxiv:1907.00397 §6]. |

## Implementation sketch

The loop is the same for any Q-function. Only `q_values` changes between
variants. Values in comments are the paper's, not general advice.

```pseudocode
# Deep Q-learning with replay and a target copy  [alg. 1] [arxiv:1907.00397 §2.3]
# Input: env, gamma, epsilon; Output: trained parameters
def train(env, params, gamma, epsilon):
    # Second copy of the same parameters, used only for targets  [§4]
    target_params = copy(params)
    # Replay capacity: 80 for frozen lake, 1000 for radio  [§4]
    memory = replay_memory(capacity=80)
    steps_since_sync = 0
    for episode in range(500):                    # [§6.2]
        state = env.reset()
        done = False
        while not done:
            # Random action with probability epsilon, else greedy  [alg. 1]
            action = choose_action(state, params, epsilon)
            next_state, reward, done = env.step(action)
            memory.add(state, action, reward, next_state, done)
            state = next_state
            steps_since_sync += 1
            # Minibatch of 5 once the memory holds more than 5  [§4]
            if len(memory) > 5:
                batch = memory.sample(5)
                targets = []
                for t in batch:
                    # Terminal: reward alone  [alg. 1]
                    if t.done:
                        targets.append(t.reward)
                    else:
                        # Best next value under the target copy  [§2.3] [§4]
                        # Double DQN variant [arxiv:2412.02574 §4.1]:
                        # pick with params, score with target_params
                        best = max(q_values(t.next_state, target_params))
                        targets.append(t.reward + gamma * best)
                predictions = [q_values(t.state, params)[t.action]
                               for t in batch]
                # Squared TD error, RMSprop with learning rate 0.01  [eq. 3] [§6.2]
                params = rmsprop_step(mse(targets, predictions), params)
            # Copy the main parameters into the target every 20 steps  [§4]
            if steps_since_sync >= 20:
                target_params = copy(params)
                steps_since_sync = 0
        # Decay epsilon: per episode for frozen lake  [eq. 16]
        # per step for radio  [eq. 17]
        epsilon = decay(epsilon)
    return params
```

Settings the paper does not state: the discount factor [arxiv:1907.00397 §2],
the frozen-lake episode length [arxiv:1907.00397 §3.1], and the weight and
bias initialisation [arxiv:1907.00397 alg. 1]. Pick these and report them.

## Contradictions

- **Target parameters.** The text says the target uses a separate target
  network [arxiv:1907.00397 §2.3] [arxiv:1907.00397 §4]. Algorithm 1 uses the
  current parameters for the target value [arxiv:1907.00397 alg. 1].
- **Target in the hardware survey.** Its eq. 3 takes the maximum over the
  target network [arxiv:2610.28699 eq. 3] [arxiv:2610.28699 §3.4]. The caption
  of its fig. 4 takes the argmax with the main network and scores it with the
  target network, which is the Double DQN form [arxiv:2610.28699 fig. 4]. The
  two are different updates. This page does not settle which one a given
  design uses.
- **Noisy radio convergence.** The text puts the optimum at about 110
  iterations [arxiv:1907.00397 §6.4]. Another passage says the reward is
  "very close to 100 in fewer than 10 iterations" [arxiv:1907.00397 §6.6].
- **Noiseless radio convergence.** The caption of the noiseless figure says
  "several iterations" for 2 and 5 channels [arxiv:1907.00397 fig. 8]. The
  text says about 100 [arxiv:1907.00397 §6.4].
- **Discount factor in the CLIP paper.** Its table gives 0.95 for DQN and 0.99
  for PPO [arxiv:2412.16201 tab. 1]. The card records that the released DQN
  code uses 0.99, so one value may be a transcription error. The paper does
  not say which.
- **Paper and its released code.** These are differences between the paper
  and its code, not between papers. The 1907 code syncs the target every 21
  steps, not 20, and takes a non-textbook exploration step. It uses 4 channels
  for the radio task whatever n is. Gamma is 0.999 in the code only. The
  code's radio observation uses a time index that the paper does not specify.
  The cards record the code details, and the paper's values are the ones
  shown above.

## Papers

- [Variational Quantum Circuits for Deep Reinforcement Learning](/papers/1907.00397):
  replaces the Q-network with a 4-qubit, 2-layer variational circuit and
  reports simulated results on frozen lake and a radio channel task.
- [Generating Critical Scenarios for Testing Automated Driving Systems](/papers/2412.02574):
  a Double DQN with prioritized replay sets scene parameters so that an Apollo
  7.0 vehicle crashes in LGSVL.
- [CLIP-RLDrive: Human-Aligned Autonomous Driving via CLIP-Based Reward Shaping in Reinforcement Learning](/papers/2412.16201):
  a CNN DQN with a CLIP-based reward term for an unsignalized intersection in
  highway-env.
- [Reward Machine Inference for Robotic Manipulation](/papers/2412.10096):
  one DQN per inferred reward-machine state for simulated block tasks.
- [Research on reinforcement learning based warehouse robot navigation algorithm in complex warehouse layout](/papers/2411.06128):
  names DQN only as a baseline.
- [Reinforcement learning for hardware acceleration and system-level optimization on FPGA and integrated-circuit platforms: A survey](/papers/2610.28699):
  a review, with no experiments of its own, of RL on FPGAs and integrated
  circuits. Its DQN section gives the loss and the hardware mapping.