Variational Quantum Circuits for Deep Reinforcement Learning

In plain words

Reinforcement learning teaches an agent to pick actions by trying them and keeping track of which ones pay off. A common recipe, deep Q-learning, uses a neural network to estimate the value of each action in each situation. This paper asks whether a small quantum circuit can do that job instead. Quantum computers available today are noisy and can only run short circuits, so the circuit has to stay small.

The circuit's settings are the knobs the learner tunes, much as neural-network weights are tuned. The state is written as a row of bits, and each bit turns one qubit on or off. A few layers of entangling gates and adjustable rotations then mix these bits. The readout for each qubit is a number, and the agent takes the action whose readout is largest. The authors test this on a small maze game and on a wireless channel-picking task. Everything runs on a simulator, and a short trial on a real quantum computer is reported too.

The appeal is the small number of tunable settings. The authors also argue that the method could be run on near-term hardware, but the experiments are small and the results are the authors' own.

TL;DR

• Deep Q-learning with experience replay and a target network, where the Q-function is a 4-qubit, 2-layer variational circuit §4 alg. 1.
• The state is encoded in computational-basis bits; the action is the qubit with the largest Pauli-Z expectation, after adding a trainable bias §4.1 §6.1.
• Frozen lake uses 28 trainable parameters where a Q-table has 64 entries §6.6; the authors report 0.9 total reward after 198 episodes §6.4.
• Cognitive radio reaches a total reward of 100 in simulation for 2 to 5 channels §6.4. Hardware runs use 5 or 7 episodes only §6.5.
• The official repository matches the frozen-lake setup closely but implements only the 4-channel radio case §6.2 NS3_Huck.py.

Problem

Q-learning needs a table with one entry per state-action pair, and the table grows fast with the state space §2.3. Deep Q-learning replaces the table with a parametric approximator. The authors note that training such approximators is unstable, because consecutive samples are correlated and the regression target moves with the parameters they update §2.3. The paper adds two fixes from the earlier DQN work: experience replay and a separate target network §2.3.

Current noisy quantum hardware cannot run deep circuits, because errors grow with the number of gates §1 §6.6. The authors therefore restrict the study to small problems. The frozen-lake task has 16 states and 4 actions §3.1. The cognitive-radio task has n² states when there are n channels, because the primary user cycles through the channels with period n §3.2 §4.2. The main experiments are simulated on a classical computer, and the authors state that the number of qubits must stay within what a simulator can handle §4 §6.6.

Method

The agent keeps two copies of the same circuit. The main copy is updated at every step; the target copy is replaced by a copy of the main one every 20 steps §4. Training draws random minibatches of 5 transitions from a replay memory with 80 slots for frozen lake and 1000 for cognitive radio §4.

Encoding. A state index is written in binary, most significant bit first, and each bit is one input qubit. Bit 1 applies a rotation by π about x and then about z, which maps |0⟩ to |1⟩; bit 0 applies no rotation §4.1 eq. 4 eq. 7 eq. 8 eq. 9 eq. 10 eq. 11 eq. 12 eq. 13 eq. 14. The encoding is the same for the cognitive-radio circuits, with one qubit per bit of the state index §4.2.

Variational layers. Each layer has a chain of CNOT gates that entangles neighbouring qubits, followed by one general single-qubit rotation with three angles on each qubit §4.1 fig. 5 eq. 15. The layer repeats twice in every experiment §4.1 §4.2. The three angles per rotation are the trained parameters §4.1.

Readout and action. The circuit returns the Pauli-Z expectation value of each qubit, estimated from repeated measurements on hardware. A trainable bias vector, one entry per qubit, is added to these values. The action is the index of the largest value, so qubit 0 maps to the first action and so on §6.1. For frozen lake, the indices map to left, down, right and up. For cognitive radio, index i selects channel i+1 §6.1.

Learning. Q-values are trained with the squared temporal-difference error from DQN eq. 3. The optimiser is RMSprop with learning rate 0.01 §6.2. The exploration rate starts at 1.0 and decays per episode in frozen lake and per step in cognitive radio §6.2 eq. 16 eq. 17.

# Q-value read-out of the variational circuit  [§4.1] [fig. 5]
# Input:  state index (int), weights (2, 4, 3), bias (4,)
# Output: Q-values (4,)
def q_values(state_index, weights, bias):
    # Write the state index as 4 bits, most significant first  [§4.1]
    bits = binary_digits(state_index, width=4)     # (4,) in {0, 1}
    # Bit 1 flips a qubit to |1>, using pi-rotations  [eq. 7] [eq. 8]
    for qubit in range(4):
        rotate_x(pi * bits[qubit], qubit)
        rotate_z(pi * bits[qubit], qubit)
    # Two layers, each with an entangling chain and rotations  [fig. 5]
    for layer in range(2):
        # CNOT chain 0->1, 1->2, 2->3  [fig. 5]
        for qubit in range(3):
            cnot(control=qubit, target=qubit + 1)
        # One general rotation with 3 angles per qubit  [eq. 15]
        for qubit in range(4):
            general_rotation(weights[layer, qubit])
    # Pauli-Z expectation per qubit, plus trainable bias  [§6.1] [§4.1]
    expectations = [expect_z(qubit) for qubit in range(4)]
    return expectations + bias

# Variational quantum deep Q-learning  [alg. 1] [§2.3] [§4]
# Input: environment, gamma, epsilon; Output: trained weights and bias
def train_vqdqn(environment, gamma, epsilon):
    # Two parameter sets with the same layout  [§4]
    weights = initialise_weights()        # scheme not stated [§4]
    bias = zeros(4)                       # one per qubit [§4.1]
    target_weights = copy(weights)
    target_bias = copy(bias)
    # Replay capacity: 80 (frozen lake), 1000 (radio)  [§4]
    memory = replay_memory(capacity=80)
    steps_since_sync = 0
    for episode in range(500):            # [§6.2]
        state = environment.reset()
        done = False
        while not done:
            # Random action with probability epsilon, else greedy on
            # the circuit read-out  [alg. 1] [§6.1]
            if random_uniform() < epsilon:
                action = random_action()
            else:
                action = argmax(q_values(state, weights, bias))
            next_state, reward, done = environment.step(action)
            memory.add(state, action, reward, next_state, done)
            state = next_state
            steps_since_sync += 1
            # Minibatch of 5 once the memory holds more than 5  [§4]
            if len(memory) > 5:
                batch = memory.sample(5)
                targets = []
                for transition in batch:
                    # Terminal: target is the reward alone  [alg. 1]
                    if transition.done:
                        target = transition.reward
                    else:
                        # Otherwise add the best next value from the
                        # target parameters  [§2.3] [§4]
                        next_values = q_values(
                            transition.next_state,
                            target_weights,
                            target_bias,
                        )
                        target = transition.reward + gamma * max(next_values)
                    targets.append(target)
                predictions = [
                    q_values(t.state, weights, bias)[t.action] for t in batch
                ]
                # RMSprop step on the mean squared error  [eq. 3] [§6.2]
                weights, bias = rmsprop_step(
                    mean_squared_error(targets, predictions), weights, bias
                )
            # Copy the main parameters into the target every 20 steps  [§4]
            if steps_since_sync >= 20:
                target_weights = copy(weights)
                target_bias = copy(bias)
                steps_since_sync = 0
        # Frozen lake: decay per episode  [eq. 16]
        # Radio: multiply by 0.99 per step instead  [eq. 17]
        epsilon = epsilon / (episode / 100 + 1)
    return weights, bias

Two points in the second block differ from the paper text, see Paper vs code. The frozen-lake and radio variants differ in the epsilon schedule, the replay capacity and the state encoding; the released code shows both.

Hyperparameters

Results

Author-reported. Simulation results use the setups in §6.2; the table gives the setup for each row.

The paper reports the frozen-lake and radio curves as mean and standard deviation over the last 100 episodes fig. 7 fig. 8. Hardware runs are inference only, with the trained parameters from simulation and 1024 shots per expectation value §6.5.

Paper vs code

Code: official repository linked from the Data Availability section, at commit 982efde0919d5ed365b15ae618b60f219dbee417. Python, PennyLane and PyTorch; frozen lake in Code/QML_DQN_FROZEN_LAKE.py, radio in Code/QML_DQN_NS3.py with Code/NS3_Huck.py as the simulator §5.

• Frozen-lake observation. The paper says the observation is the record of all time steps §3.1. The code encodes only the current grid cell index in 4 bits QML_DQN_FROZEN_LAKE.py QML_DQN_FROZEN_LAKE.py.
• Target in Algorithm 1. Algorithm 1 uses the current parameters θ for the target value y_j alg. 1. The text says the target uses a separate target network θ⁻ §2.3 §4. The code uses the target network QML_DQN_FROZEN_LAKE.py.
• Exploration rule. The paper describes ε-greedy with random action probability ε alg. 1. The code takes the greedy action when a uniform draw is below ε/|A| + (1 − ε); otherwise it draws uniformly over all |A| actions, which can also be greedy QML_DQN_FROZEN_LAKE.py. With 4 actions and ε = 1, greedy is chosen with probability 0.25 + 0.75 × 0.25 = 0.4375, not 0.25.
• Target sync. The text says every 20 steps §4. The code syncs when the counter exceeds 20, which is every 21 steps. The counter runs across episodes QML_DQN_FROZEN_LAKE.py QML_DQN_FROZEN_LAKE.py.
• Discount and frozen-lake step limit. The paper gives no value for gamma or for the frozen-lake episode length §2 §3.1. The code sets gamma to 0.999 and allows 2500 steps QML_DQN_FROZEN_LAKE.py.
• Unused learning-rate argument. The code passes alpha = 0.4 but never uses it; the optimiser's learning rate is 0.01 QML_DQN_FROZEN_LAKE.py QML_DQN_FROZEN_LAKE.py. The paper's RMSprop value is 0.01 §6.2.
• Episode bookkeeping. Epsilon decay and reward logging run only when an episode ends with done. An episode that hits the step limit is not logged and does not decay epsilon QML_DQN_FROZEN_LAKE.py.
• Training start. The code trains only once the memory holds more than 5 transitions QML_DQN_FROZEN_LAKE.py. The paper says to sample a minibatch each step alg. 1.
• Gradients. The paper points to gradient estimation for expectation values on quantum hardware §4. The code uses exact autodiff through PennyLane's torch interface on the simulator QML_DQN_FROZEN_LAKE.py QML_DQN_FROZEN_LAKE.py. The hardware runs are inference only §6.5, and the repository has no hardware or noise code. Qiskit appears only in commented-out device lines QML_DQN_FROZEN_LAKE.py QML_DQN_NS3.py.
• Radio simulator. The paper says observations are ns3 statistics §3.2. The code uses a Python class, Radio, that imitates a single primary user cycling through channels; the file header cites ns-3 gym NS3_Huck.py NS3_Huck.py. Reward and episode termination match tab. 1 and §3.2 NS3_Huck.py.
• Radio channel count. The paper reports 2, 3, 4 and 5 channels and a 2-channel noisy run §6.4. The code builds Radio() with its default of 4 channels and sets n_actions = 4 QML_DQN_NS3.py NS3_Huck.py. The circuit has 4 qubits in every case QML_DQN_NS3.py. Trained parameters for the 2-channel noisy run are in the repository, but no training code for it is VQDQN_NS3_2_Channels_IBM_AER_WITH_NOISE.
• Radio state. The observation is (channel status, time) §4.2. The code encodes the occupied channel times the channel count plus the time index read after the step's increment QML_DQN_NS3.py QML_DQN_NS3.py. The paper does not say which time index is used.
• Radio randomness. The radio code reseeds NumPy from the wall clock on every exploration decision QML_DQN_NS3.py. The paper speaks of a reproducible setting fig. 8. Runs are not reproducible from the code.
• Radio epsilon. The code multiplies epsilon by 0.99 at every step, as the paper says §6.2 eq. 17 QML_DQN_NS3.py.

Pitfalls

• Algorithm 1 and the text disagree on the target parameters (see Paper vs code). Implement the target network, as the code does, and state the choice.
• Exploration as the code writes it is not textbook ε-greedy (see Paper vs code). Use the textbook rule unless you want to match the released run.
• Weight initialisation is not stated in the paper. The code uses 0.01 times a standard normal for the rotations and zeros for the bias QML_DQN_FROZEN_LAKE.py.
• Q-values are the Pauli-Z expectation values plus a bias §6.1 QML_DQN_FROZEN_LAKE.py QML_DQN_FROZEN_LAKE.py. Nothing bounds the bias, so Q-values are not forced into the reward range. The paper does not discuss the scale.
• Gamma is not stated §2. The code value, 0.999, is in the repository only. Treat the reported numbers as specific to that value.
• Frozen-lake episode length is not stated. The code uses 2500 steps QML_DQN_FROZEN_LAKE.py.
• Radio episode length is 100 steps and ends after 3 collisions §3.2. The released code's radio observation uses the time index after increment (see Paper vs code). Keep that index if you want to match the results.
• Convergence claims disagree within the paper. Radio, noisy runs: optimum at about 110 iterations §6.4 against "very close to 100 in fewer than 10 iterations" §6.6. Radio, noiseless: the caption of fig. 8 says "several iterations" for 2 and 5 channels, while the text says about 100 §6.4. Treat the figure captions as the less precise source.
• Frozen-lake "iteration" and "episode" are used interchangeably §6.4 fig. 7.
• The hardware evidence is 5 radio episodes and 7 frozen-lake episodes, with no variance reported tab. 3 tab. 4. Do not read it as a benchmark.
• Simulation cost grows with qubit count. The paper's experiments use at most 5 qubits, and the authors say scaling beyond n ≤ 5 is untested §6.6.
• The "quantum advantage" is a parameter count, not a speed or accuracy claim. The radio neural-network count comes from the ns3-gym paper, as the authors cite it §6.6. It was not re-checked here.
• The noise study uses a single 2019 device snapshot §6.3 app. Noise Information.
• NS3_Huck.py uses np.int NS3_Huck.py NS3_Huck.py, an alias NumPy removed in version 1.24. The released radio code will not run on current NumPy without changes. The paper gives no version numbers for PennyLane, PyTorch or Qiskit.

Lineage

• Builds on: deep Q-learning with experience replay and a target network, as introduced by Mnih et al. (2015) §2.3; the variational circuit approach and encoding of Mitarai et al. (2018) §1 §4. Neither is a card here.
• Compared against: a tabular Q-learning table with 64 entries for frozen lake §6.6; the neural network from the ns3-gym work for the radio task §6.6.
• Superseded by: not stated in the paper.