Reinforcement learning for hardware acceleration and system-level optimization on FPGA and integrated-circuit platforms: A survey

In plain words

Reinforcement learning (RL) is a way for a program to learn by trial and error, guided by rewards. Chips whose circuits can be reprogrammed (FPGAs) or that are built once for one job (custom integrated circuits) can run that learning with less delay and power than a general-purpose processor, but they are much harder to program. This review collects more than a hundred studies that tried it. It sorts them by the learning method used, by whether RL is the thing being sped up or the thing that tunes the hardware, and by application area such as networking, security, robots, vehicles and sensing. It ends with the problems that remain open.

TL;DR

• A literature survey, not a method. It runs no experiments of its own §1.2.
• It covers more than 100 FPGA and integrated-circuit RL studies, grouped by RL family, research role and application §1 §3 §4 §5.
• Its tables list author-reported best results. The paper itself warns that baselines differ between studies, so the numbers cannot be ranked against each other §3.9.
• Several internal inconsistencies (study count, one speedup, a few table values) should be checked against the cited sources before reuse; see Pitfalls.

Problem

Existing reviews cover only part of the field: some stop at FPGAs, some at one algorithm family, and some only name RL as future work. The paper argues that no single review compares all major RL families across both FPGA and integrated- circuit implementations §1.1 tab. 1.

Method

This is a review, so the "method" is its literature selection and organisation.

Search and selection §1.2. Scopus was the main index. The Boolean query requires an RL term (RL, DRL, "reinforcement learning", "deep reinforcement learning") in the title, abstract or keywords, combined with an FPGA or IC term (FPGA, ASIC, IC, SoC, "integrated circuit", and related). Filters: English, journal article or conference paper, years 2015 to 2026. Excluded: redundant studies, papers without enough hardware detail, papers without evaluation results, and papers only weakly related to RL on FPGA/IC. Older work already covered by earlier surveys was kept only where it supports the taxonomy. The paper gives no counts for retrieved, screened or included records.

Organisation. The text is organised along three axes:

1. RL family, with hardware notes for each family §3. Section 3 presents the RL families in the order MDP §3.1, Q-learning §3.2, SARSA §3.3, deep Q-learning §3.4, DDPG §3.5, actor-critic §3.6, Monte Carlo §3.7, PPO §3.8, and a normalised-metrics subsection §3.9. Multi-agent RL is covered under research areas §4.1.2.
2. Research role, where RL is either the workload that the hardware runs §4.1 or the optimizer that tunes the hardware §4.2. Sub-areas are listed in tab. 6.
3. Application domain: networking §5.1, hardware and embedded security §5.2, robotics §5.3, autonomous vehicles §5.4, intelligent sensing and imaging §5.5, other domains §5.6, and an electric-vehicle charging case study §5.7. A cross-domain table is tab. 7; a per-application results table is tab. 8.

Background on FPGA and IC resources, and the design-flow levels from algorithm modelling down to physical design, is in §2.1 and tab. 2. Simulators and platforms that decouple RL from the hardware toolchain are in §2.2.

Equations the paper states for each family. Notation follows the paper. Tables and equations are cited by the paper's own numbers.

• Tabular Q-learning update eq. 1 §3.2.
• SARSA update, which uses the action actually taken next instead of the maximum eq. 2 §3.3.
• Deep Q-learning loss over replay minibatches, with a target network eq. 3 §3.4.
• Actor-critic sequence on hardware eq. 4, TD error eq. 5 §3.6.
• A3C, a set of asynchronous workers whose gradients are accumulated into shared parameters eq. 6; the one-step advantage each worker uses eq. 7 §3.6.
• SAC sequence eq. 8 and objective with an entropy term eq. 9 §3.6.
• PPO clipped surrogate objective, shown as an unnumbered display §3.8.

# Tabular Q-learning update  [eq. 1] [§3.2]
# Input:  table q of shape (S, A); one transition (s, a, r, s_next)
# Output: the updated table q

def q_update(q, s, a, r, s_next, alpha, gamma):
    # Greedy estimate of the best value reachable from s_next  [eq. 1]
    best_next = max(q[s_next])
    # Temporal-difference error against the stored value  [eq. 1]
    td_error = r + gamma * best_next - q[s, a]
    # Move the stored value a step of size alpha towards the target  [eq. 1]
    q[s, a] = q[s, a] + alpha * td_error
    return q

# Deep Q-learning loss on a replay minibatch  [eq. 3] [§3.4]
# Input:  minibatch of (s, a, r, s_next) drawn from the replay buffer
# Output: scalar loss

def dqn_loss(minibatch, online_net, target_net, gamma):
    # Accumulate the squared TD error over the batch  [eq. 3]
    squared_errors = []
    for s, a, r, s_next in minibatch:
        # Target uses the slowly updated target network  [eq. 3]
        best_next_value = max(target_net(s_next))
        td_target = r + gamma * best_next_value
        # Prediction of the online network for the taken action  [eq. 3]
        prediction = online_net(s)[a]
        squared_errors.append((td_target - prediction) ** 2)
    return mean(squared_errors)

PPO's clipped surrogate, with ratio r_t(\theta) and advantage A_t, is \mathbb{E}[\min(r_t(\theta) A_t, \mathrm{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) A_t)] §3.8, unnumbered display. The paper names GAE as the advantage estimator used with PPO but does not define it §3.8.

Hardware mapping, as the paper describes it.

• Q-learning: the table lives in BRAM or LUTRAM; the max over actions becomes a comparator tree; exploration (epsilon-greedy or Boltzmann) becomes a small state machine §3.2. SARSA drops the max comparator, so its datapath is simpler and its update latency does not grow with the number of actions §3.3.
• DQN: forward passes map to pipelined matrix-vector units; the replay buffer sits in BRAM or DDR; the target network is a second module updated at a slower rate §3.4 fig. 4. Training is split into stages in some designs §3.4.
• DDPG: a deterministic actor replaces the argmax, so there is no comparator tree; the critic takes the concatenated state and action; exploration noise is injected at the actor output §3.5 fig. 5.
• Actor-critic and A3C: actor and critic share pipelines; the TD error feeds both update paths; A3C uses replicated worker pipelines and a global update unit §3.6 eq. 4 eq. 5 eq. 6.
• Monte Carlo: no critic and no TD error; a return-accumulation block and a trajectory buffer replace them §3.7.
• PPO: needs a ratio block, a clip-and-select stage, rollout buffers and repeated minibatch passes; the clip and ratio logic can dominate run time more than the network layers §3.8.
• MARL: the paper groups accelerators by bottleneck, which are control latency, inter-agent communication and deep-network compute, not by algorithm §4.1.2.

Architectural patterns the paper extracts §4.1.1 §7. On-chip replay buffers, pipelined learning engines, parallel environment execution, fixed-point arithmetic and dataflow-oriented network implementations recur across designs. The paper concludes that no single RL paradigm dominates and that algorithm choice has to follow both the task and the hardware budget §7.

Hyperparameters

The survey does not train anything, so it states almost no training hyperparameters. The values below are the numeric hardware and training parameters the paper does give, and the gaps an implementer would need to fill from the original works.

Per-design data widths (N), action counts (A) and clocks are in tab. 4.

Results

The survey reports no new results. The figures below are as the cited works report them, which the paper repeats in tab. 4 and tab. 8. Baselines differ by study (CPU, GPU, prior design, or none), so the values are not comparable across rows §3.9.

Table 4 as printed. Columns follow the paper. BRAM, LUT, LUTRAM, FF and DSP are resource use in percent; N is the number of bits; A is the action count; CLK is in MHz; PW is power in W; IL is the implementation level. "Limitations" is omitted here, and the paper's own remarks are cited in Pitfalls.

Table 8 as printed (selected columns). Tasks are short paraphrases; the paper's grouping of rows into domains is not consistent in its table layout, so the rows are kept in printed order. "--" means the paper gives no value.

Paper vs code

No official code.

The TeX source, abstract and arXiv metadata contain no code or repository link. The paper links no project page. It is a literature survey with no method of its own to implement, so no GitHub search was run. Repositories that the survey cites for individual works are not used.

Pitfalls

• Study count disagrees across the paper. The abstract and conclusion say "more than one hundred" §7. The introduction says "more than 120" §1, and Table 1 lists 120 tab. 1. The conclusion also says "more than two decades" §7, but the search window is 2015 to 2026 §1.2. Treat the count as approximate.
• No screening numbers. The search is described, but the paper gives no record counts at each stage, so the selection cannot be reproduced §1.2.
• PPOAccel speedup differs. Table 4 says up to 26× throughput tab. 4; the PPO section says 27× §3.8. Check the original paper.
• Missing values in the text. The FA3C improvement §3.6 and the Ge et al. speedup §3.6 are left blank in the text. The Table 4 entry for Ge et al. says 2.39× over GPU, which the text does not repeat.
• Possibly shifted columns in Table 4. The da2018parallel row lists a clock of 26.42 MHz, the same figure as its throughput. The bazmalah2026resource row has N = 1024 and PW = 36 with 0.35% LUT use. The ge2025e2cap row has N = 512. The FARANE-Q row lists PW = 67 with no unit note. The abbreviation list defines N as number of bits tab. 4. These values look inconsistent with their own abbreviations; check them against the cited sources before use.
• DQN target written two ways. Eq. 3 uses the target network's maximum eq. 3 §3.4. Fig. 4's caption uses an argmax taken from the main network inside the target, which is the double-DQN form fig. 4. The two are not the same update.
• Advantage and GAE. The A3C advantage in eq. 7 is a one-step TD error [eq. 7]. The PPO section says GAE is used, but GAE and its lambda are not defined §3.8. Do not assume the two are the same estimator.
• SAC objective simplified. Eq. 9 writes the objective as Q minus alpha times log-probability over sampled pairs eq. 9. It does not show the state distribution or the twin-critic minimum, and the temperature update is not given §3.6.
• Typo that changes meaning. The DDPG section says "FPGA introduces a differentiable deterministic actor" §3.5. The author means DDPG.
• Normalised metrics are defined but not applied. Table 5 defines energy per operation, FIPS/W, resource efficiency and energy-delay product tab. 5. The survey does not compute them for any design, apart from quoting the FIPS/W value for one TD3 design §3.9 tab. 4.
• Speedups are not comparable. The baselines are CPU, GPU, another FPGA design, or none. The paper says so itself §3.9. Do not rank rows of Table 4 or Table 8 against each other.
• Table 8 has unclear entries. Some rows show negative values without a unit or baseline, such as −999 ms for one inference-time comparison tab. 8. Others give a "+6×" beside a "48×" with no stated meaning tab. 8. Read the source paper before quoting them.
• Hardware-vs-algorithm claims are qualitative. The "real-time capability" column of the algorithm comparison is judgement, not measurement tab. 3.
• Figures are described, not reproduced. The paper's figures (roadmap, architecture diagrams, EV charging case) are cited by number. They are not included here, under the licensing rules for this site.

Lineage

• Builds on: the same author's earlier RL taxonomy, cited for the RL and DRL distinction §3. No card exists for it.
• Compared against: four earlier surveys on RL or ML for FPGA and reconfigurable hardware tab. 1. None has a card.
• Superseded by: not known from this wiki.