Deep Q-learning

Overview

Q-learning keeps one value per state-action pair. That table grows with the state space, so deep Q-learning replaces it with a function that has trainable parameters 1907.00397 §2.3. The authors note that this kind of training is unstable, because consecutive samples are correlated and the regression target moves as the parameters change 1907.00397 §2.3.

Two fixes recur: experience replay, which trains on random stored transitions, and a target copy of the parameters, which supplies the bootstrapped value 1907.00397 §2.3 1907.00397 §4. Most cards use both, but not all. 2412.10096 §5 keeps a buffer per machine state and does not state a target network. 2411.06128 §VII-B names DQN only as a baseline and gives no details. The hardware survey describes both parts as well 2610.28699 §3.4.

The update is the squared temporal-difference error between the current value of the action taken and the reward plus the best discounted value under the target copy, with a terminal state using the reward alone 1907.00397 eq. 3 1907.00397 alg. 1. Variants change the target. Double DQN lets the evaluation network choose the next action and the target network score it 2412.02574 §4.1. Other cards take the maximum over the target copy 2412.16201 eq. 5. The loss also differs: squared error in 1907.00397 eq. 3 and 2412.16201 eq. 4, Huber in 2412.10096 eq. 1. The discount factor is not stated in the 1907 paper 1907.00397 §2.

Variants

Variational quantum circuit as the Q-function

The state index is written in binary and each bit sets one qubit with a π-rotation. Two layers follow, each with a CNOT chain and one general single-qubit rotation with three trained angles per qubit. The read-out is the Pauli-Z expectation of each qubit plus a trainable bias, and the action is the largest entry 1907.00397 §4.1 1907.00397 fig. 5 1907.00397 §6.1. Frozen lake needs 24 rotation angles and 4 bias values, 28 in total 1907.00397 §4.1 1907.00397 tab. 2. The paper's classical comparison uses a neural network from the ns3-gym work, cited with a parameter count of 2n² + 2n 1907.00397 §6.6. That count is the authors' citation and was not checked. Card: Variational Quantum Circuits for Deep Reinforcement Learning.

Double DQN with prioritized replay

The agent chooses scene changes for a driving simulator. The evaluation network picks the next action and the target network scores it 2412.02574 §4.1. Transitions are drawn in proportion to stored priorities, which the paper ties to how well the selection network scored each action. It gives no values for the priority exponents 2412.02574 §4.1. The Q-network has two hidden ReLU layers and one output per action; hidden sizes are not stated 2412.02574 §4.1. Exploration starts at ε = 1.0 and falls to 0.1 2412.02574 §4.1. Card: Generating Critical Scenarios for Testing Automated Driving Systems.

Convolutional Q-network with a reward term

The Q-network is a CNN over four stacked frames 2412.16201 fig. 2 2412.16201 §2.1. Its one-step target uses a periodically copied target network 2412.16201 §2.1 2412.16201 eq. 5. The paper adds a reward term from CLIP, weighted by 1.2 2412.16201 §2.4. The agent drives an unsignalized left turn. Author-reported, the DQN with that term reaches 96% success with one other vehicle, against 70% for DQN without it 2412.16201 tab. 2. The same table has plain PPO at 78% 2412.16201 tab. 2. Card: CLIP-RLDrive: Human-Aligned Autonomous Driving via CLIP-Based Reward Shaping in Reinforcement Learning.

One Q-function per reward-machine state

Each state of the inferred reward machine has its own Q-function and its own experience buffer. The buffers also store demonstrations, and all Q-functions are updated from sampled batches at the end of each episode 2412.10096 §5. The paper uses the DQRM approach for this 2412.10096 §5. The loss is the Huber TD loss 2412.10096 eq. 1 2412.10096 eq. 2. The Q-values are pixel-wise maps over pick and place locations 2412.10096 §5 2412.10096 fig. 2. Results are shown as learning curves only 2412.10096 fig. 5. Card: Reward Machine Inference for Robotic Manipulation.

Baseline only

The warehouse paper reports a DQN accuracy ratio of 89.76 against 96.42 for its own method 2411.06128 tab. II. The DQN architecture, training and hyperparameters are not stated 2411.06128 §VII-B, so the figure says little about DQN as a method. Card: Research on reinforcement learning based warehouse robot navigation algorithm in complex warehouse layout.

Hardware accelerator for DQN

The survey treats DQN as a workload that FPGA or integrated-circuit hardware runs. It writes the same squared loss over replay minibatches, with a target network 2610.28699 eq. 3 2610.28699 §3.4. In the hardware mapping it describes, forward passes run in pipelined matrix-vector units. The replay buffer sits in on-chip block RAM or in DDR memory, and the target network is a second module updated at a slower rate 2610.28699 §3.4 2610.28699 fig. 4. Some designs split training into stages 2610.28699 §3.4. The survey gives no target-update period. It notes one design that uses minibatches of about 1 to 8 2610.28699 §3.4.

The survey's table of DQN designs lists reported results. Examples are 19.05 times faster than a GPU for one Virtex-7 design, and 671 times faster at inference for an Ultra96-V2 design whose baseline the survey does not state 2610.28699 tab. 4. The survey says baselines differ between studies, so these values cannot be ranked against each other 2610.28699 §3.9. Some table rows look inconsistent with the survey's own column definitions, so check a cited work before quoting one 2610.28699 tab. 4. The designs it cites have no cards here, and nothing from them is used. Card: Reinforcement learning for hardware acceleration and system-level optimization on FPGA and integrated-circuit platforms: A survey.

Settings in the cards

Values as each card records them. A row shows one paper's choices, not a tested effect: no card compares these variants on the same task.

When to use which

The cards support the rows below. Each row says what its card shows.

Implementation sketch

The loop is the same for any Q-function. Only q_values changes between variants. Values in comments are the paper's, not general advice.

# Deep Q-learning with replay and a target copy  [alg. 1] [arxiv:1907.00397 §2.3]
# Input: env, gamma, epsilon; Output: trained parameters
def train(env, params, gamma, epsilon):
    # Second copy of the same parameters, used only for targets  [§4]
    target_params = copy(params)
    # Replay capacity: 80 for frozen lake, 1000 for radio  [§4]
    memory = replay_memory(capacity=80)
    steps_since_sync = 0
    for episode in range(500):                    # [§6.2]
        state = env.reset()
        done = False
        while not done:
            # Random action with probability epsilon, else greedy  [alg. 1]
            action = choose_action(state, params, epsilon)
            next_state, reward, done = env.step(action)
            memory.add(state, action, reward, next_state, done)
            state = next_state
            steps_since_sync += 1
            # Minibatch of 5 once the memory holds more than 5  [§4]
            if len(memory) > 5:
                batch = memory.sample(5)
                targets = []
                for t in batch:
                    # Terminal: reward alone  [alg. 1]
                    if t.done:
                        targets.append(t.reward)
                    else:
                        # Best next value under the target copy  [§2.3] [§4]
                        # Double DQN variant [arxiv:2412.02574 §4.1]:
                        # pick with params, score with target_params
                        best = max(q_values(t.next_state, target_params))
                        targets.append(t.reward + gamma * best)
                predictions = [q_values(t.state, params)[t.action]
                               for t in batch]
                # Squared TD error, RMSprop with learning rate 0.01  [eq. 3] [§6.2]
                params = rmsprop_step(mse(targets, predictions), params)
            # Copy the main parameters into the target every 20 steps  [§4]
            if steps_since_sync >= 20:
                target_params = copy(params)
                steps_since_sync = 0
        # Decay epsilon: per episode for frozen lake  [eq. 16]
        # per step for radio  [eq. 17]
        epsilon = decay(epsilon)
    return params

Settings the paper does not state: the discount factor 1907.00397 §2, the frozen-lake episode length 1907.00397 §3.1, and the weight and bias initialisation 1907.00397 alg. 1. Pick these and report them.

Contradictions

• Target parameters. The text says the target uses a separate target network 1907.00397 §2.3 1907.00397 §4. Algorithm 1 uses the current parameters for the target value 1907.00397 alg. 1.
• Target in the hardware survey. Its eq. 3 takes the maximum over the target network 2610.28699 eq. 3 2610.28699 §3.4. The caption of its fig. 4 takes the argmax with the main network and scores it with the target network, which is the Double DQN form 2610.28699 fig. 4. The two are different updates. This page does not settle which one a given design uses.
• Noisy radio convergence. The text puts the optimum at about 110 iterations 1907.00397 §6.4. Another passage says the reward is "very close to 100 in fewer than 10 iterations" 1907.00397 §6.6.
• Noiseless radio convergence. The caption of the noiseless figure says "several iterations" for 2 and 5 channels 1907.00397 fig. 8. The text says about 100 1907.00397 §6.4.
• Discount factor in the CLIP paper. Its table gives 0.95 for DQN and 0.99 for PPO 2412.16201 tab. 1. The card records that the released DQN code uses 0.99, so one value may be a transcription error. The paper does not say which.
• Paper and its released code. These are differences between the paper and its code, not between papers. The 1907 code syncs the target every 21 steps, not 20, and takes a non-textbook exploration step. It uses 4 channels for the radio task whatever n is. Gamma is 0.999 in the code only. The code's radio observation uses a time index that the paper does not specify. The cards record the code details, and the paper's values are the ones shown above.

Papers

• Variational Quantum Circuits for Deep Reinforcement Learning: replaces the Q-network with a 4-qubit, 2-layer variational circuit and reports simulated results on frozen lake and a radio channel task.
• Generating Critical Scenarios for Testing Automated Driving Systems: a Double DQN with prioritized replay sets scene parameters so that an Apollo 7.0 vehicle crashes in LGSVL.
• CLIP-RLDrive: Human-Aligned Autonomous Driving via CLIP-Based Reward Shaping in Reinforcement Learning: a CNN DQN with a CLIP-based reward term for an unsignalized intersection in highway-env.
• Reward Machine Inference for Robotic Manipulation: one DQN per inferred reward-machine state for simulated block tasks.
• Research on reinforcement learning based warehouse robot navigation algorithm in complex warehouse layout: names DQN only as a baseline.
• Reinforcement learning for hardware acceleration and system-level optimization on FPGA and integrated-circuit platforms: A survey: a review, with no experiments of its own, of RL on FPGAs and integrated circuits. Its DQN section gives the loss and the hardware mapping.