AP-DRL: A Synergistic Algorithm-Hardware Framework for Automatic Task Partitioning of Deep Reinforcement Learning on Versal ACAP

Enlai Li; Zhe Lin; Sharad Sinha; Wei Zhang

cs.ARMar 31, 2026

AP-DRL: A Synergistic Algorithm-Hardware Framework for Automatic Task Partitioning of Deep Reinforcement Learning on Versal ACAP

Authors: Enlai Li, Zhe Lin, Sharad Sinha, Wei Zhang

Organizations: The Hong Kong University of Science and Technology, Hong Kong SAR, China · Sun Yat-Sen Univeristy, Guangdong, China · Indian Institute of Technology Goa, Goa, India

Abstract

Deep reinforcement learning has demonstrated remarkable success across various domains. However, the tight coupling between training and inference processes makes accelerating DRL training an essential challenge for DRL optimization. Two key issues hinder efficient DRL training: (1) the significant variation in computational intensity across different DRL algorithms and even among operations within the same algorithm complicates hardware platform selection, while (2) DRL's wide dynamic range could lead to substantial reward errors with conventional FP16+FP32 mixed-precision quantization. While existing work has primarily focused on accelerating DRL for specific computing units or optimizing inference-stage quantization, we propose AP-DRL to address the above challenges. AP-DRL is an automatic task partitioning framework that harnesses the heterogeneous architecture of AMD Versal ACAP (integrating CPUs, FPGAs, and AI Engines) to accelerate DRL training through intelligent hardware-aware optimization. Our approach begins with bottleneck analysis of CPU, FPGA, and AIE performance across diverse DRL workloads, informing the design principles for AP-DRL's inter-component task partitioning and quantization optimization. The framework then addresses the challenge of platform selection through design space exploration-based profiling and ILP-based partitioning models that match operations to optimal computing units based on their computational characteristics. For the quantization challenge, AP-DRL employs a hardware-aware algorithm coordinating FP32 (CPU), FP16 (FPGA/DSP), and BF16 (AI Engine) operations by leveraging Versal ACAP's native support for these precision formats. Comprehensive experiments indicate that AP-DRL can achieve speedup of up to 4.17 $\times$ over programmable logic and up to 3.82 $\times$ over AI Engine baselines while maintaining training convergence.

Figures & tables

Fig. 1: Workflow overview of the DRL algorithm.

Fig. 2: The overview of Versal ACAP (AI Edge series) architecture.

Fig. 3: The binary format of BF16, FP16 and FP32.

Fig. 4: Log-normalized execution time of three DRL algorithm-environment combinations for single training timestep on PS, PL, and AIE-ML processors on the VEK280 platform (hardware emulation), with performance profiled by the Vitis Analyzer. Detailed configurations of these three combinations can be found in Table III .

Fig. 5: Execution time breakdown of one single training timestep on VEK280 PS (hardware emulation), with performance profiled through the xiltimer APIs. The configuration details of the three DRL algorithm-environment combinations can be found in Table III .

Fig. 6: Execution time breakdown of the synthetic GEMMs on VEK280 PL and AIE-ML (hardware emulation), with performance profiled by the Vitis Analyzer. The clock frequencies of PL and AIE are 245 MHz and 1 GHz, respectively.

Fig. 7: The overview of the AP-DRL framework.

Fig. 8: The network architecture and FLOPs of forward propagation and backpropagation in the DQN-Breakout environment.

Pragma	Configurations	#Design Points
Dataflow (DF)	Enable/Disable	2
Function Pipeline (FP)	Enable/Disable	2
Loop Pipeline (LP)	Enable/Disable	2
Loop Unroll (LU)	Factors	$\lceil\log_{2}LB\rceil$
Array Partition (AP)	Types & Factors	$B_{M}/B_{D}+1$

TABLE I: Design Points for PU Candidates in TAPCA

	FP16	FP32	BF16
Binary Format (Sign, Exp, Frac)	(1, 5, 10)	(1, 8, 23)	(1, 8, 7)
Exponent Range (Base-2)	[-14, 15]	[-126, 127]	[-126, 127]
Fraction Precision (Decimal Digital)	7 bits ( $\sim$ 3)	10 bits ( $\sim$ 7)	23 bits ( $\sim$ 2)
Memory Usage (Bytes/Parameter)	2	4	2
Master Weight Backup Required?	Yes	N/A	No
Loss Scaling Required?	Yes	N/A	No

TABLE II: Comparison of FP16, FP32 and BF16 in Mixed-Precision Training

Fig. 9: MTP(FP16,BF16/FP32) with the whole training steps in Algorithm 1 : Mixed-precision training workflow in FP16+BF16/FP32. Techniques of master weight backup and dynamic loss scaling are used.

Fig. 10: The hardware implementation of the quantization algorithm.

Environment	Algorithm	Environment Space $\left(\lvert\mathcal{S}\rvert,\lvert\mathcal{A}\rvert\right)$	Network Architecture	Train FLOPs (Per Batch Size)	Reward Error (%)
CartPole	DQN	$(4,2)$	3-layer MLP Hidden layer 1 size: 64 Hidden layer 2 size: 64	28.04 K	1.60
InvPendulum	A2C	$(4,1)$	3-layer MLP Hidden layer 1 size: 64 Hidden layer 2 size: 64	2.31 K	4.81
LunarCont	DDPG	$(8,2)$	3-layer MLP Hidden layer 1 size: 400 Hidden layer 2 size: 300	2.25 M	1.73
MntnCarCont	DDPG	$(2,1)$	3-layer MLP Hidden layer 1 size: 400 Hidden layer 2 size: 300	2.19 M	1.12
Breakout	DQN	$(84\times 84\times 4,4)$	ConvNet+MLP Conv1(4, 32, krnl_size=8, stride=4) Conv2(32, 64, krnl_size=4, stride=2) Conv3(64, 64, krnl_size=3, stride=1) FC1(3136, 512), FC2(512, 4)	68.17 M	1.25
MsPacman	PPO	$(84\times 84\times 4,9)$	ConvNet+MLP Conv1(4, 32, krnl_size=8, stride=4) Conv2(32, 64, krnl_size=4, stride=2) Conv3(64, 64, krnl_size=3, stride=1) FC1(3136, 512), FC2(512, 9)	106.23 M	1.13

TABLE III: Experiment Configurations and Results

Fig. 11: Convergence performance and average reward of AP-DRL. Solid curves represent the mean values across 5 independent runs with different random seeds, while shaded regions indicate the corresponding standard deviations.

Network Architecture (Hidden Size)	Training Time (FP32)	Training Time (BF16)	Speedup
3-Layer MLP (64, 64)	0.21 ms	0.27 ms	0.78 $\times$
3-Layer MLP (400, 300)	32.55 ms	28.81 ms	1.13 $\times$
3-Layer MLP (4096, 3072)	2175.12 ms	729.91 ms	2.98 $\times$

TABLE IV: Training Time in One Episode for DQN-CartPole Using AP-DRL Across Network Architectures.

Fig. 12: Normalized execution time of AIE-only, FIXAR, and AP-DRL across environments, algorithms, and batch sizes. Normalization is performed separately between DQN-CartPole and A2C-InvPendulum, DDPG-LunarCont and DDPG-MntnCarCont, and DQN-Breakout and PPO-MsPacman.

Fig. 13: Normalized training throughput of AIE-only, FIXAR, and AP-DRL across environments, algorithms, and batch sizes. Normalization is performed separately between DQN-CartPole and A2C-InvPendulum, DDPG-LunarCont and DDPG-MntnCarCont, and DQN-Breakout and PPO-MsPacman.

Fig. 14: The operation sequence of DDPG-LunarCont in AP-DRL within one single timestep. The batch size of DDPG-LunarCont is 256.

Fig. 15: The partitioning results of AP-DRL on DDPG-LunarCont with batch sizes of 256, 512, and 1024. Each node represents a matrix multiplication layer (fully-connected layer in DDPG-LunarLander).

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

Apr 14, 2026

TCL: Enabling Fast and Efficient Cross-Hardware Tensor Program Optimization via Continual Learning

Deep learning (DL) compilers rely on cost models and auto-tuning to optimize tensor programs for target hardware. However, existing approaches depend on large offline datasets, incurring high collection costs and offering suboptimal transferability across platforms. In this paper, we introduce TCL, a novel efficient and transferable compiler framework for fast tensor program optimization across diverse hardware platforms to address these challenges. Specifically, TCL is built on three core enablers: (1) the RDU Sampler, a data-efficient active learning strategy that selects only 10% of tensor programs by jointly optimizing Representativeness, Diversity, and Uncertainty, substantially reducing data collection costs while maintaining near-original model accuracy; (2) a new Mamba-based cost model that efficiently captures long-range schedule dependencies while achieving a favorable trade-off between prediction accuracy and computational cost through reduced parameterization and lightweight sequence modeling; and (3) a continuous knowledge distillation framework that effectively and progressively transfers knowledge across multiple hardware platforms while avoiding the parameter explosion and data dependency issues typically caused by traditional multi-task learning. Extensive experiments validate the effectiveness of each individual enabler and the holistic TCL framework. When optimizing a range of mainstream DL models on both CPU and GPU platforms, TCL achieves, on average, 16.8x and 12.48x faster tuning time, and 1.20x and 1.13x lower inference latency, respectively, compared to Tenset-MLP.

Chaoyao Shen, Linfeng Jiang, Yixian Shen +5

Southeast University, China · University of Amsterdam, Netherlands · Southeast University & Shandong Yunhai Guochuang Innovative Technology Co., Ltd., China

Data-Efficient Learning

Ask PDF

Oct 14, 2026

Deep Reinforcement Learning for HPC Job Scheduling: A Statistical Evaluation

Deep Reinforcement Learning (DRL) offers learned job selection for High-Performance Computing (HPC), but performance against heuristics alone does not establish value beyond action masking. Six DRL treatments are compared: Proximal Policy Optimization (PPO), Advantage Actor-Critic (A2C), and Deep Q-Network (DQN), each with and without masking. Three heuristics and a seed-paired masked uniform random control provide baselines across ten seeds on two real Slurm (Simple Linux Utility for Resource Management) workload traces, one homogeneous and one heterogeneous, under a common non-parametric framework. All masked treatments complete evaluation, whereas unmasked treatments are truncated. On the homogeneous trace, no masked DRL treatment significantly improves average waiting time or average and maximum bounded slowdown over the control. On the heterogeneous trace, MaskablePPO improves all three metrics relative to the control. Its average waiting time is equivalent to that of the three heuristics within a 10% margin, and its maximum bounded slowdown is 22% lower than that of Last-Come, First-Served (LCFS). MaskablePPO has the leading average waiting-time rank among complete treatments on both traces, but Nemenyi comparisons detect no significant difference from MaskableA2C. Among masked variants, MaskablePPO has the lowest recorded mean training time, while MaskableDQN has the lowest mean decision latency. These findings distinguish completion from baseline-specific scheduling gains and support including a masked random control when assessing the value of learned selection.

Justin M. Cheney, Clement N. Nyirenda

Computer Science Department, University of the Western Cape, Robert Sobukwe Rd, Bellville, Cape Town, 7535, South Africa · eResearch Office, University of the Western Cape, Bellville, Cape Town, South Africa

Reinforcement Learning High-Performance Computing Deep Q-Learning

Ask PDF

Apr 6, 2026

Vehicle-as-Prompt: A Unified Deep Reinforcement Learning Framework for Heterogeneous Fleet Vehicle Routing Problem

Unlike traditional homogeneous routing problems, the Heterogeneous Fleet Vehicle Routing Problem (HFVRP) involves heterogeneous fixed costs, variable travel costs, and capacity constraints, rendering solution quality highly sensitive to vehicle selection. Furthermore, real-world logistics applications often impose additional complex constraints, markedly increasing computational complexity. However, most existing Deep Reinforcement Learning (DRL)-based methods are restricted to homogeneous scenarios, leading to suboptimal performance when applied to HFVRP and its complex variants. To bridge this gap, we investigate HFVRP under complex constraints and develop a unified DRL framework capable of solving the problem across various variant settings. We introduce the Vehicle-as-Prompt (VaP) mechanism, which formulates the problem as a single-stage autoregressive decision process. Building on this, we propose VaP-CSMV, a framework featuring a cross-semantic encoder and a multi-view decoder that effectively addresses various problem variants and captures the complex mapping relationships between vehicle heterogeneity and customer node attributes. Extensive experimental results demonstrate that VaP-CSMV significantly outperforms existing state-of-the-art DRL-based neural solvers and achieves competitive solution quality compared to traditional heuristic solvers, while reducing inference time to mere seconds. Furthermore, the framework exhibits strong zero-shot generalization capabilities on large-scale and previously unseen problem variants, while ablation studies validate the vital contribution of each component.

Shihong Huang, Shengjie Wang, Lei Gao +4

Polytechnic Institute, Zhejiang University, 310015, Hangzhou, China · S.F. Technology Co., Ltd., 518054, Shenzhen, China · School of Management, Zhejiang University, 310058, Hangzhou, China

Vehicle Routing RL for Combinatorial Optimization Neural Combinatorial Optimization Reinforcement Learning

Ask PDF

Abstract

Reinforcement Learning

Explore similar work

TCL: Enabling Fast and Efficient Cross-Hardware Tensor Program Optimization via Continual Learning

Deep Reinforcement Learning for HPC Job Scheduling: A Statistical Evaluation

Vehicle-as-Prompt: A Unified Deep Reinforcement Learning Framework for Heterogeneous Fleet Vehicle Routing Problem