cs.LGOct 20, 2026

Reinforcement learning for hardware acceleration and system-level optimization on FPGA and integrated-circuit platforms: A survey

Authors: Hamza Kheddar

Organizations: LSEA Laboratory, Department of Electrical Engineering, University of Medea, 26000, Algeria

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Reinforcement learning (RL) enables adaptive and autonomous decision-making in embedded, edge, and heterogeneous computing systems. Field-programmable gate arrays (FPGAs) and integrated circuits (ICs) offer low latency, energy efficiency, and architectural flexibility for RL deployment, while introducing challenges in resource constraints, implementation complexity, and hardware-algorithm co-design. This review surveys more than 100 studies on RL implementations across FPGA and IC platforms, emphasizing architectures, implementation strategies, performance evaluation, and applications. The literature is organized by RL family, including model-based Markov decision processes, classical value-based methods, deep RL, actor-critic, policy-gradient, and multi-agent approaches. We further classify studies by abstraction level, hardware realization strategy, and application domain, examining how RL algorithms map onto reconfigurable and fixed hardware. Unlike surveys restricted to particular algorithms or platforms, this work provides a unified analysis of FPGA and IC implementations, covering acceleration, reconfigurable architectures, energy-aware control, and system-level optimization. Comparative assessments consider resource utilization, latency, throughput, power consumption, and reported performance gains. Finally, we identify recurring architectural patterns, practical limitations, open challenges, and research directions for efficient RL-driven hardware systems. The survey offers a consolidated reference for researchers developing and evaluating RL solutions on specialized hardware.

Figures & tables

Ref.PaperNoPsRelatedICFPGABGDRL typesResearchRLCaseLimitations 
 typeRLwork   MDPQLSARSADQNDDPGACMCPPOareasApps.studyPerspectives 
[ 118 ]Survey14○○●◐○●○●●○○●○○○◐ 
[ 44 ]Survey9●○●●○○○○○○○○○◐○◐ 
[ 162 ]Review1○◐●◐○○○○○○○○○◐○◐ 
[ 99 ]Review1○●●○○○○○○○○○○○○◐ 
OurReview120●●●●●●●●●●●●●●●● 
Table 1: Comparison of the proposed review with existing FPGA-related surveys and reviews. A mark (●) signifies that a specific area has been covered, whereas a mark (○) denotes that it has not been addressed. The symbol (◐) indicates that key aspects of the field remain largely unaddressed.
Figure 1: Survey roadmap outlining the organization of RL -based FPGA and IC platforms.
Figure 2: Overview of two hardware architecture representations that could be employed for RL workloads. (a) A simplified FPGA floorplan showing the distribution of on-chip resources—CLB logic tiles, BRAM columns, global routing spines, configuration/control logic, global buffers, clock management blocks, and I/O banks—with a CLB slice highlighting LUT, FF, LUTRAM, and carry logic elements. (b) A conceptual IC / SoC architecture illustrating functional subsystems relevant to RL execution, including the CPU core, hardware accelerator, on-chip memory, interconnect (Bus/NoC), power management, clock management, and I/O interfaces and pads.
Design LevelFPGA WorkflowIC / ASIC Workflow
SLDAlgorithm modeling, dataflow simulation, partitioning (MATLAB/Simulink, SystemC, Python, PYNQ)Same high-level modeling and system exploration (MATLAB/Simulink, SystemC, C/C++)
DSCRL graph-level optimization, operator fusion, quantization, memory tiling, dataflow scheduling, accelerator mapping; possible generation of HLS / RTL or accelerator intermediate representationSimilar graph/dataflow-level compilation targeting ASIC / SoC accelerators; supports early exploration of data reuse, memory hierarchy, and hardware mapping
HLSC/C++/SystemC →\rightarrow RTL (Vitis HLS, Intel HLS, CatapultC); directives for unrolling, pipelining, memory partitioningC/C++/SystemC →\rightarrow RTL (Cadence Stratus, Mentor Catapult, Synopsys HLS); early power/performance design-space exploration
RTLVerilog/VHDL/SystemVerilog RTL for datapaths, FSM ( FSM )s, controllers; synthesizable design for LUT , FFs, DSP s, BRAMsVerilog/SystemVerilog RTL for logic synthesis, optimization, timing-driven transformations
Gate-LevelFPGA-specific placement, routing, timing closure (Vivado, Quartus); bitstream generation for predefined fabricASIC physical design including floorplanning, placement, routing, CTS, STA, IR drop analysis, DRC/LVS; GDSII tape-out
Table 2: Unified abstraction levels for FPGA and IC design flows
TermMeaning
aaCurrent action aa at time tt
ssPresent state ss at time tt
st+1s_{t+1}Next state observed after taking aa in state ss
at+1a_{t+1}Next action taken in state st+1s_{t+1} following the policy being used.
γ\gammaThe discount factor, which scales the importance of future rewards. ( 0<γ<10<\gamma<1 )
α\alphaThe learning rate, determining how much the new information should influence the existing Q-value.
Table 3: Notation
ModelProblem scenario suitable forAdvantagesLimitationsTypical applicationsReal-time capability
MDPModel-based control with known or identifiable dynamics; small/medium discrete S,AS,AInterpretable, analyzable; formal guarantees on optimal policy and safetyRequires accurate transition model; poor scalability, mainly offline planningTest-configuration generation, interconnect coverage, coarse DVFS or power-state selectionLow–moderate (offline planning, light online lookup)
QLDiscrete, low-dimensional state/action spaces; no explicit model availableSimple tabular updates; easy to implement in fixed-point hardwareExplodes with large S,AS,A ; slow exploration and convergenceMode selection for DVFS , cache / memory policy, reconfiguration of small regionsModerate (for compact tables in BRAM)
SARSAStreaming control tasks requiring smooth policy updates; tasks sensitive to abrupt action changesStable incremental learning; updates follow executed behavior; well-suited for hardware-friendly TD ( TD ) updatesStill tabular with limited scalability; performance depends strongly on exploration strategyThermal/power throttling, adaptive subsystem tuning, sequential reconfiguration tasksModerate (lightweight updates suitable for real-time streams)
DQNHigh-dimensional state, discrete actions; rich sensor/monitor signals on FPGAHandles large SS with deep nets; good for complex policiesTraining cost, replay memory, stability and quantization issues on-chipPlacement / routing guidance, task scheduling, NoC routing, adaptive accelerator tuningHigh for inference (training mostly off-chip)
DDQNSame as DQN where value overestimation hurts reliabilityReduces Q-overestimation; more stable value estimatesStill heavy training; needs two networks and memory resourcesLatency–power trade-off control, adaptive mapping of kernels, thermal-aware controlHigh for inference (training offline, batched)
ACMixed continuous/discrete control; need both policy and value estimatesSupports continuous actions; good sample efficiency vs. pure policy gradientCoupled AC training; sensitive to hyperparametersFine-grained DVFS , analog front-end tuning, throughput–latency controlModerate–high (inference light, training heavier)
Table 4: Refined comparison of RL - based FPGA algorithms covering key applications
               
RefRLHardwareResource consumption (%)NACLKPWILBest resultLimitations
 Used BRAMLUTSLUTRAMFFDSP      
[ 82 ]MDPZynq-7000–15.35–1.85–8–100–HLS7.8 ×\times accelerationHLS MDP generator; limited scope; no BRAM/DSP data
[ 89 ]QLZynq UltraScale+ MPSoC–0.0560.0100.0120.231842400.038RTLUsage: -50% in resourceLimited to small action spaces, offline training, no large-scale benchmarks.
[ 132 ]QLZynq UltraScale+ MPSoC–0.080.030.030842220.037RTL–Uses small Q-tables, limited exploration strategies, simplified multiplier logic, and no dynamic environment testing.
[ 135 ]QLPYNQ-Z124.299.771.906.3403241001.75RTL1400 ×\times speedupLimited state modeling, no multi-intersection support, and high BRAM usage for large environments.
Table 5: Overview of representative RL - based FPGA / IC proposed schemes, highlighting hardware platforms, resource consumption, implementation level, performance results, and limitations.
Figure 3: High-level architecture of the QL agent mapped to FPGA / IC hardware. The QL accelerator executes the tabular update rule using DSP -based MAC units and fixed-point arithmetic, with Q-values stored in BRAM/URAM. The learning rate α\alpha governs the magnitude of the parameter update, while the discount factor γ\gamma scales the contribution of future reward terms. The PG , implemented with LUT logic and a lightweight pseudo-random module, produces the next action at+1a_{t+1} to support stochastic exploration. Temporal elements z−1z^{-1} act as register-based pipeline delays to synchronize sts_{t} , ata_{t} , and rtr_{t} in hardware. The architecture highlights a practical mapping of QL operators to FPGA resources ( DSP , LUT , BRAM/URAM) without enforcing a device-specific implementation [ 132 ] .
Figure 4: High-level architecture of a DQN agent on FPGA / IC . The Main Q-network is implemented using DSP arrays and fixed-point MAC units, with parameters stored in BRAM/URAM. Experience replay is mapped to BRAM/URAM or external DDR depending on capacity. The Target Q-network is maintained as a BRAM mirror and updated via a soft-update unit ( DSP + FSM ). The loss L=(Q−y)2L=(Q-y)^{2} is computed using the TD target y=r+γQtarget(st+1,arg⁡max⁡aQ(st+1,a))y=r+\gamma Q_{\text{target}}(s_{t+1},\arg\max_{a}Q(s_{t+1},a)) .
Figure 5: High-level functional architecture of a DDPG agent. The Main Actor generates continuous control actions, with exploration noise injected during interaction with the environment. The Main Critic evaluates state–action pairs and computes the TD target using the Target Actor and Target Critic, whose parameters are updated through soft (Polyak) averaging. A replay buffer enables off-policy learning by storing and supplying past transitions (st,at,rt,st+1)(s_{t},a_{t},r_{t},s_{t+1}) for both actor and critic updates. State–action concatenation indicates where state and action vectors are combined prior to critic evaluation. Solid lines denote forward dataflow paths used during online interaction and gradient-based learning, whereas dashed lines indicate auxiliary or delayed pathways used exclusively for target value computation and parameter synchronization. Target networks are passive, auxiliary models not used for online control, and are invoked only during TD-target evaluation and soft updates. Learning hyperparameters (e.g., α\alpha , γ\gamma , τ\tau ) are applied during loss computation and update rules and are not explicit network inputs.
MetricExpressionInterpretationPreferred trend
Throughput-based energy efficiencyEET=T/P\mathrm{EE_{T}=T/P}Measures the amount of useful RL computation achieved per watt. The throughput T\mathrm{T} may represent inferences/s, updates/s, samples/s, or full iterations/s, depending on the reported workload.Higher
FIPS/WFIPS/P\mathrm{\mathrm{FIPS}/P}Measures full RL iterations processed per watt. This metric is particularly suitable for accelerators that execute both decision-making and learning-update stages.Higher
Energy per operationEop=P/T\mathrm{E_{\mathrm{op}}=P/T} or Eop=P×L\mathrm{E_{\mathrm{op}}=P\times L}Estimates the energy consumed by one inference, update, sample, or full RL iteration. It is useful when comparing low-power and real-time accelerators.Lower
Average resource footprintRUavg=1∣R∣∑r∈RUr\mathrm{RU_{\mathrm{avg}}=\frac{1}{|\mathcal{R}|}\sum_{r\in\mathcal{R}}U_{r}}Represents the mean utilization of the reported FPGA resources, where R\mathcal{R} may include BRAM , LUT , LUTRAM , FF , and DSP usage.Lower
Bottleneck resource utilizationRUmax⁡=max⁡r∈RUr\mathrm{RU_{\max}=\max_{r\in\mathcal{R}}U_{r}}Identifies the most heavily used hardware resource. This metric helps reveal scalability limits even when the average resource usage appears moderate.Lower
Resource efficiencyRET=T/RUavg\mathrm{RE_{T}=T/RU_{\mathrm{avg}}}Relates useful RL throughput to average resource utilization. It helps compare designs that achieve similar performance with different hardware costs.Higher
Table 6: Comprehensive normalized metrics for cross-design comparison of RL -based FPGA / IC implementations.
 Research AreaPrimary FocusTypical ObjectivesKey Challenges
RL as the WorkloadRL-based accelerationMapping RL computation (value updates, NN inference, gradients) onto FPGA pipelinesReduce latency; increase throughput; exploit parallelism; support on-device trainingResource constraints; precision tuning; balancing compute pipelines
 Multi-Agent RLFPGA -based architectures supporting distributed or local MARL agentsScalable learning; cooperative control; reduced centralized bottlenecksNon-stationarity; coordination overhead; gradient synchronization
 Multi-FPGA RL platformsDistributed RL execution across multiple interconnected FPGA sLarge-scale RL; higher throughput; extended memory capacityCommunication latency; consistency of updates; partitioning strategies
 Energy-Efficient RL AccelerationEnergy-efficient RL execution on FPGA using neuromorphic, mixed-precision, and hierarchical designsReduce energy consumption of RL learning/inference; improve GOP/s/W; enable low-power on-device adaptationTask specificity; limited online learning flexibility; memory and precision trade-offs
 HW–SW co-explorationJoint exploration of algorithmic and hardware design parameters using RL -based searchAutomated design-space exploration; precision selection; mapping and scheduling optimizationLarge search spaces; long exploration time; generalization to unseen workloads
RL as the OptimizerReconfig. RL approachesRuntime reconfiguration controlled by RL agents (DPR, adaptive tuning)Adaptive performance; robustness under workload/environment variations; real-time controlReconfiguration cost; policy stability; ensuring safe reconfiguration
Table 7: Overview of the two principal roles of RL in FPGA / IC research and their associated sub-domains, highlighting the primary focus, typical objectives, and key challenges of each area.
Figure 6: Unified RL -assisted synthesis workflow covering both HLS and logic-synthesis stages. RL agents interact with several points in the design flow: Task (1) optimizing HLS transformations; Task (2) incorporating feedback from specifications and HLS reports; Task (3) guiding logic optimization; and Task (4) assisting FPGA and ASIC technology mapping. This illustrates how RL provides an adaptive optimization layer across the full IC/ FPGA synthesis pipeline.
Figure 7: Application domains of RL - based FPGA and IC platforms.
App. domainTypical RL algo.Hardware implementation strategyMain objectiveKey constraints
NetworkingQL, DQN, dueling DQN, TD3, MARLFPGA-accelerated routing, CNN/RL inference, distributed Q-tables, and pipelined decision enginesLow-latency routing, spectrum access, waveform adaptation, and traffic controlDynamic traffic, scalability, communication overhead, and real-time response
Hardware and embedded securityQL, DQN, PPO, AC, RL-RNN, HDC-based RLRTL-level verification engines, FPGA-based detection pipelines, HDC/RL accelerators, and adaptive test enginesVulnerability detection, intrusion detection, side-channel analysis, and security verificationLarge state spaces, adversarial behavior, verification complexity, and limited generalization
RoboticsQL, DQN, AC, DDPGReal-time FPGA control pipelines, fixed-point policy inference, and hardware-in-the-loop controlFast feedback control, adaptive behavior, and RL training accelerationStrict timing, limited resources, control stability, and sensor uncertainty
Autonomous vehiclesPPO, AC, DQN, MARLCPU–GPU–FPGA task placement, FPGA-accelerated policy inference, and hardware-in-the-loop validationSelf-driving control, UAV tracking, trajectory optimization, and energy-aware operationSafety, latency, energy consumption, and multi-objective optimization
Intelligent sensing and imagingDeep RL, PDS-RL, SAC, PG methodsLightweight accelerators, stochastic computing, FPGA–CPU hybrid control, and adaptive sensing pipelinesFeature selection, signal reconstruction, adaptive acquisition, and sensing optimizationSensor noise, low-power operation, real-time imaging, and data uncertainty
Other applicationsQL, DQN, model-free RLFPGA-based inference/control cores, edge-training pipelines, and safety-filtered decision enginesMotor torque control, DNA sequence alignment, power converter control, and embedded optimizationSafety constraints, domain specificity, limited benchmarks, and deployment reliability
Table 8: Cross-domain comparison of RL algorithm selection and FPGA/IC implementation strategies.
Apps.Ref.FPGA usedRLDedicated task and limitationsCompared toResultImprovement
 [ 24 ]Nexys A7 / PYNQ-Z2QLProposes a QL-based fault- and congestion-aware NoC routing framework with a compact Q-table; however, it relies on offline training, synthetic evaluations, and lacks formal deadlock guarantees.HARAQ, RELAR0.95 delivery rate at 20% faults–
 [ 76 ]PXIe-7975REINFORCEEmploys RL to develop a flexible duplex system, targeting maximization of area spectral efficiency. However, the system’s reliance on the OP-map for transmission decisions limits it in dynamic environments.Full duplex OP-mapBetter spectral efficiency–
Networking[ 83 ]Zynq Z7020Model-free RLThe work targeted accelerated RL - based routing in delay-tolerant networks using FPGA, but faced scalability constraints, and communication overhead.No RL (CPU)Response Time ≈\approx 2.5s Throughput ≈\approx 41 bundles/s×\times 32 faster
 [ 32 ]Xilinx xc7z045ffg900-2TD3Real-time waveform synthesis improves resilient wireless signal classification, but faces complexity, FPGA usage, training instability, and sensitivity to channel noise.No-FIR DeepRadioIDAcc. (High SNR) =100%4.1 ×\times (No-FIR) 1.9 ×\times (DeepRadioID)
 [ 75 ]PYNQ-Z2Dueling- DQNEnabled real-time Dueling- DQN routing on LEO satellites using FPGA-accelerated CNNs. But faced higher design complexity, resource limits, and scalability constraints.CPU - onlyInf. time =0.3 s3.10 ×\times faster
 [ 60 ]Kintex-7 FPGA KC705QL-MARLIt learned adaptive fault-tolerant routing paths in mesh NoCs, but required per-router Q-tables and added hardware overhead with higher resource usage.QL–based routingPacket delivery +7.1%2.7% fewer hardware resources and 0.89% less power
Table 9: Summary of well-investigated FPGA-based RL applications, including FPGA boards used, dedicated tasks, limitations, and performance improvements over baseline methods.
Figure 8: Operational flow of AdaTest, an RL -based hardware Trojan ( HT ) detection framework [ 19 ] . Adaptive test patterns generated by the agent are applied to the CUT , and the resulting responses are compared with expected outputs: mismatches indicate a Trojaned circuit, while matching responses classify the design as Benign .
Figure 9: RL - based Internet of vehicles model. The vehicle primarily consisting of a CPU, GPU, and FPGA placement model, is trained on a cloud server with the objective of reducing both execution latency and energy consumption using RL [ 164 ] .
Figure 10: RL-based FPGA EV charging management system: (a) overall FPGA architecture; (b) RL engine; (c) policy engine with LFSR-based exploration; (d) state discretizer; (e) action generator; (f) reward generator [ 27 ] .
More in this topic

GPU-Accelerated RL

See all 15 papers

Explore similar work

CardsList
Sep 26, 2026

All On-Board: Fully On-Chip Neuromorphic Q-Learning with Embedded CartPole Simulation

As AI models grow in size and usage, their energy demands increase dramatically, raising sustainability and economic concerns. Neuromorphic hardware, inspired by the energy efficiency of the brain, seeks to address this challenge by offering low-power, fast-processing alternatives to conventional computing. Such hardware is particularly well-suited to control systems deployed in resource-constrained environments, which are best trained via reinforcement learning (RL). This contribution presents the design and implementation of a fully on-chip, closed-loop Loihi 2 RL agent. Our neuromorphic circuit consists of a fully embedded Q-learning algorithm and an on-chip simulation of the CartPole-v0 environment on Loihi 2. Our Q-learning algorithm trained the same number of successful agents as the CPU implementation in only half the execution time and with two orders of magnitude less dynamic power. These findings demonstrate the viability of RL on neuromorphic hardware and highlight its promise for building energy-efficient, real-time, embedded AI systems.

Steven C. Nesbit, Giovanni T. Michel, Gerd J. Kunde +2

College of Computing and Informatics, Drexel University, Philadelphia, PA, USA · Information Sciences (CAI-3), Los Alamos National Laboratory, Los Alamos, NM, USA · Department of Electrical and Computer Engineering, Northwestern University, Evanston, IL, USA +1

Neuromorphic ComputingQ-Learning
AskPDF
Feb 25, 2026

Architectural Design and Performance Analysis of FPGA based AI Accelerators: A Comprehensive Review

Deep learning (DL) has emerged as a rapidly developing advanced technology, enabling the performance of complex tasks involving image recognition, natural language processing, and autonomous decision-making with high levels of accuracy. However, as these technologies evolve and strive to meet the growing demands of real-life applications, the complexity of DL models continues to increase. These models require processing of massive volumes of data, demanding substantial computational power and memory bandwidth. This gives rise to the critical need for hardware accelerators that can deliver both high performance and energy efficiency. Accelerator types include ASIC based solutions, GPU accelerators, and FPGA based implementations. The limitations of ASIC and GPU accelerators have led to FPGAs becoming one of the prominent solutions, offering distinct advantages for DL workloads. FPGAs provide a flexible and reconfigurable platform, allowing model specific customization while maintaining high efficiency. This article explores various hardware level optimizations for DL. These optimizations include techniques such as loop pipelining, parallelism, quantization, and various memory hierarchy enhancements. In addition, it provides an overview of state-of-the-art FPGA-based neural network accelerators. Through the study and analysis of these accelerators, several challenges have been identified, paving the way for future optimizations and innovations in the design of FPGA-based hardware accelerators.

Soumita Chatterjee, Sudip Ghosh, Tamal Ghosh +1

Indian Institute of Engineering Science and Technology, Shibpur, India

FPGA AccelerationFPGA-Based Neural Network AccelerationEfficient Neural Network InferenceEnergy-Efficient ML
AskPDF
Apr 27, 2026

BitRL: Reinforcement Learning with 1-bit Quantized Language Models for Resource-Constrained Edge Deployment

The deployment of intelligent reinforcement learning (RL) agents on resource-constrained edge devices remains a fundamental challenge due to the substantial memory, computational, and energy requirements of modern deep learning systems. While large language models (LLMs) have emerged as powerful architectures for decision-making agents, their multi-billion parameter scale confines them to cloud-based deployment, raising concerns about latency, privacy, and connectivity dependence. We introduce BitRL, a framework for building RL agents using 1-bit quantized language models that enables practical on-device learning and inference under severe resource constraints. Leveraging the BitNet b1.58 architecture with ternary weights (-1, 0, +1) and an optimized inference stack, BitRL achieves 10-16x memory reduction and 3-5x energy efficiency improvements over full-precision baselines while maintaining 85-98 percent of task performance across benchmarks. We provide theoretical analysis of quantization as structured parameter perturbation, derive convergence bounds for quantized policy gradients under frozen-backbone architectures, and identify the exploration-stability trade-off in extreme quantization. Our framework systematically integrates 1-bit quantized language models with reinforcement learning for edge deployment and demonstrates effectiveness on commodity hardware.

Md. Ashiq Ul Islam Sajid, Mohammad Sakib Mahmood, Md. Tareq Hasan +3

Department of Computer Science BRAC University Dhaka 1212, Bangladesh · Department of Computer Science Missouri State University Springfield, MO, USA · Department of Computer Science & Engineering Prime University Dhaka, Bangladesh +2

LLM QuantizationRL for Language ModelsOn-Device Language Model InferenceEdge MLLow-Bit Quantization
AskPDF