Role-Adaptive Policy Optimization for Offline Reinforcement Learning

Seonvin Cho; Soohyun Choi; Songnam Hong

cs.LGSep 30, 2026

Role-Adaptive Policy Optimization for Offline Reinforcement Learning

Authors: Seonvin Cho, Soohyun Choi, Songnam Hong

Organizations: Department of Electronic Engineering, Hanyang University Seoul, Republic of Korea

Abstract

Policy regularization in offline reinforcement learning balances policy improvement against reliance on uncertain value estimates. This balance can differ between selecting actions for execution and supplying actions for critic bootstrapping, yet methods such as TD3+BC couple these roles through a shared policy. We propose Role-Adaptive Policy Optimization (RAPO), which adapts policy-update coefficients according to their roles in value learning and execution. RAPO learns these coefficients by differentiating through candidate policy updates formed using the base algorithm's actor objective. For TD3+BC, RAPO separates bootstrap and execution actors and adapts their coefficients independently: the bootstrap objective penalizes policy-induced changes in target values, while the execution objective evaluates a local policy-improvement surrogate. For IQL, whose value learning is already independent of the execution actor, RAPO preserves the original value updates and adapts only the inverse temperature in advantage-weighted policy extraction. Experiments on D4RL locomotion and AntMaze tasks show improvements over both base algorithms, with larger gains for TD3+BC, whose RAPO instantiation outperforms baselines on average.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of RAPO. $\pi_{\beta}$ denotes the behavior represented by the offline data. (a) TD3+RAPO separates the bootstrap actor $\pi_{B}$ from the execution actor $\pi_{E}$ and adapts their coefficients $\alpha_{B}$ and $\alpha_{E}$ using target-stability and local-improvement objectives, respectively. (b) IQL+RAPO retains actor-independent Q/V learning from dataset transitions and adapts only the execution policy’s inverse temperature $\beta_{E}$ .

Initialize actors, coefficients, and value/target networks.
for each training step do
	Sample $\mathcal{B}_{\mathrm{in}}$ ; apply the value updates of $m$ from Section 2.2 .
	if coefficient update is due then
		Sample independent $\mathcal{B}_{\mathrm{out}}$ ; form each role’s candidate via ( 7 ).
		Update each $c_{i}$ via ( 6 ), using $\mathcal{L}_{E}$ in ( 10 ) or $\mathcal{L}_{B}$ in ( 12 ).
	Apply the scheduled actor and target updates of $m$ .

Algorithm 1 RAPO (optimizer timing and schedules in Appendix B.2 )

Dataset	TD3+BC	IQL	wPC	A2PR	ASPC	TD3+RAPO	IQL+RAPO
HalfCheetah-m	$50.0\!\pm\!0.7$	$48.4\!\pm\!0.2$	$54.7\!\pm\!0.4$	$55.9\!\pm\!0.6$	$57.8\!\pm\!0.7$	$\bm{58.2\!\pm\!0.5}$	$48.6\!\pm\!0.2$
HalfCheetah-mr	$46.2\!\pm\!0.5$	$44.2\!\pm\!0.4$	$48.2\!\pm\!0.4$	$48.0\!\pm\!1.1$	$\bm{50.1\!\pm\!1.3}$	$49.7\!\pm\!0.7$	$44.6\!\pm\!0.5$
HalfCheetah-me	$99.5\!\pm\!1.0$	$89.5\!\pm\!3.9$	$100.0\!\pm\!0.5$	$98.9\!\pm\!1.7$	$92.2\!\pm\!12.8$	$\bm{100.6\!\pm\!3.0}$	$92.8\!\pm\!0.6$
Hopper-m	$61.5\!\pm\!2.7$	$63.0\!\pm\!2.5$	$83.9\!\pm\!6.8$	$83.4\!\pm\!2.5$	$81.9\!\pm\!5.2$	$\bm{101.8\!\pm\!0.2}$	$64.4\!\pm\!5.8$
Hopper-mr	$78.8\!\pm\!34.2$	$87.6\!\pm\!8.4$	$101.2\!\pm\!0.6$	$100.1\!\pm\!1.1$	$99.6\!\pm\!4.0$	$\bm{101.5\!\pm\!0.6}$	$94.1\!\pm\!15.4$
Hopper-me	$111.4\!\pm\!2.0$	$99.8\!\pm\!24.0$	$103.5\!\pm\!7.4$	$\bm{112.6\!\pm\!0.5}$	$109.2\!\pm\!2.7$	$111.0\!\pm\!1.3$	$108.3\!\pm\!5.1$

Table 1: D4RL performance at one million training steps, reported as mean

\pm

sample standard deviation across four training seeds. Initial coefficients are fixed at five for both RAPO methods. Hyperparameters are reported in Appendix B.2 . Average rows give equal weight to environments; the final row averages all fifteen tasks. Bold marks the highest mean in each row, including ties.

Figure 2: Learned coefficients on three Hopper datasets and AntMaze-medium-play. The top row shows TD3+RAPO’s

\alpha_{B}

and

\alpha_{E}

; the bottom row shows IQL+RAPO’s

\beta_{E}

. Curves and shaded bands show means and sample standard deviations across seeds. The dotted line marks the common initial value of five.

Domain	Q	Linear	$\widehat{B}_{\pi}$ (Ours)
Locomotion	$87.0$	$88.7$	$\mathbf{91.7}$
AntMaze	$66.6$	$65.5$	$\mathbf{70.0}$
Overall	$78.8$	$79.4$	$\mathbf{83.0}$

Table 2: Execution-score ablation at the selected coefficient learning rates. Q uses target-critic value change; Linear uses the first-order gain. Entries are equal-weight task means within each domain and overall. Bold marks the highest mean.

Domain	Fixed $\alpha_{B}=5$	TD3+RAPO
Locomotion	$89.3$	$91.7$
AntMaze	$55.8$	$70.0$
Overall	$75.9$	$83.0$

Table 3: Bootstrap-coefficient adaptation with

\alpha_{E}

adapted in both conditions. The control fixes

\alpha_{B}=5

; TD3+RAPO adapts it from five. Entries are equal-weight task means within each domain and overall. Per-task results appear in Table 8 .

Domain	Fixed endpoints	TD3+RAPO
Locomotion	$88.7$	$91.7$
AntMaze	$62.9$	$70.0$
Overall	$78.4$	$83.0$

Table 4: Fixing learned endpoint coefficients from the start versus adapting from five. Both conditions retain separate TD3 actors. Scores are equal-weight task means; fixed constants and per-task results appear in Table 9 .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	TD3+RAPO	IQL+RAPO
Actor hidden layers	$2\times 256$	$2\times 256$
Value hidden layers	Q: $3\times 256$ , LayerNorm	Q and V: $2\times 256$ , no LayerNorm
Network optimizer / initial rate	Adam / $3\times 10^{-4}$	Adam / $3\times 10^{-4}$
Actor learning-rate schedule	Constant	Cosine decay
Candidate optimizer	E: current-state Adam; B: SGD	E: current-state Adam
Actual actor optimizer / interval	Adam / 2	Adam / 1

Appendix

Table 5: Training settings. Intervals are measured in value-update steps;

\bar{Q}\leftarrow(1-\tau_{Q})\bar{Q}+\tau_{Q}Q

defines the target-update rate.

Environment	TD3+RAPO	IQL+RAPO
HalfCheetah-medium	$2\times 10^{-3}$	$3\times 10^{-4}$
HalfCheetah-medium-replay	$2\times 10^{-3}$	$10^{-3}$
HalfCheetah-medium-expert	$10^{-3}$	$10^{-3}$
Hopper-medium	$2\times 10^{-3}$	$3\times 10^{-4}$
Hopper-medium-replay	$2\times 10^{-3}$	$3\times 10^{-4}$
Hopper-medium-expert	$3\times 10^{-4}$	$10^{-3}$

Appendix

Table 6: Main-selected coefficient learning rates with initialization five. TD3 uses the listed rate for both coefficients; its fixed-bootstrap control reuses the same rate for

\alpha_{E}

. Fixed-endpoint TD3 has no active coefficient update.

Environment	$3\!\times\!10^{-4}$	$10^{-3}$	$2\!\times\!10^{-3}$
HalfCheetah-medium	$54.3\pm 0.6$	$56.4\pm 0.7$	$58.2\pm 0.5$
HalfCheetah-medium-replay	$47.9\pm 0.5$	$49.3\pm 0.3$	$49.7\pm 0.7$
HalfCheetah-medium-expert	$98.1\pm 2.8$	$100.6\pm 3.0$	$98.0\pm 3.9$
Hopper-medium	$83.9\pm 11.2$	$96.4\pm 5.7$	$101.8\pm 0.2$
Hopper-medium-replay	$96.7\pm 6.4$	$99.8\pm 2.6$	$101.5\pm 0.6$
Hopper-medium-expert	$111.0\pm 1.3$	$79.1\pm 20.3$	$45.4\pm 5.5$

Appendix

Table 7: TD3+RAPO sensitivity to the initial coefficient learning rate, with coefficient initialization five. Average rows weight environments equally.

Figure 3: Fixed and adaptive bootstrap regularization at TD3+RAPO’s selected learning rates. Bars show domain or overall task averages; error bars show standard deviations across seed-wise task averages. All conditions adapt

\alpha_{E}

from five; adaptive

\alpha_{B}

starts at one or five. Per-task results appear in Table 8 .

Environment	Fixed $\alpha_{B}$		Adaptive $\alpha_{B}$
	1	5	1	5
HalfCheetah-medium	$56.6\pm 0.9$	$59.1\pm 0.6$	$58.1\pm 1.0$	$58.2\pm 0.5$
HalfCheetah-medium-replay	$48.7\pm 2.4$	$50.0\pm 0.6$	$49.1\pm 0.5$	$49.7\pm 0.7$
HalfCheetah-medium-expert	$103.1\pm 3.6$	$98.8\pm 3.7$	$95.3\pm 5.7$	$100.6\pm 3.0$
Hopper-medium	$88.5\pm 24.2$	$101.7\pm 0.5$	$101.9\pm 0.7$	$101.8\pm 0.2$
Hopper-medium-replay	$98.7\pm 1.3$	$101.3\pm 1.2$	$100.5\pm 1.5$	$101.5\pm 0.6$

Appendix

Table 8: Fixed and adaptive bootstrap regularization at the selected learning rates. All conditions retain two actors and adapt

\alpha_{E}

from five. Under Fixed, column numbers are constant

\alpha_{B}

values; under Adaptive, they are initial values.

Environment	Fixed $\alpha_{E}$	Fixed $\alpha_{B}$	Fixed endpoints	TD3+RAPO
HalfCheetah-medium	19.4	13.4	$58.3\pm 0.4$	$58.2\pm 0.5$
HalfCheetah-medium-replay	18.8	17.4	$49.9\pm 0.9$	$49.7\pm 0.7$
HalfCheetah-medium-expert	12.6	10.2	$98.8\pm 5.4$	$100.6\pm 3.0$
Hopper-medium	19.6	17.0	$101.9\pm 0.4$	$101.8\pm 0.2$
Hopper-medium-replay	23.3	16.7	$101.0\pm 1.1$	$101.5\pm 0.6$
Hopper-medium-expert	7.1	7.1	$83.8\pm 20.3$	$111.0\pm 1.3$

Appendix

Table 9: Endpoint-fixed TD3 control. Each coefficient pair averages the final values of the main runs and remains fixed throughout the control runs. Constants are displayed to one decimal place; training uses six decimal places. TD3+RAPO uses the main benchmark runs.

Explore similar work

Diffusion Policy with Behavioral Advantage Correction for Offline Reinforcement Learning
Aug 3, 2026Botao Dong, Longyang Huang, Ning Pang +1Diffusion Policies Offline Reinforcement Learning
Botao Dong is with the School of Chemistry, Chemical Engineering and Biotechnology, Nanyang Technological University, Singapore 637459
Ratio-Variance Regularized Policy Optimization
May 26, 2026Yu Luo, Shuo Han, Yihan Hu +5Frictive Policy Optimization Trust Region
Department of Foundation Model, 2012 Labs, Huawei Shanghai · Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University · Department of Computer Science and Technology, Tsinghua University +1
Low-Rank Adaptation for Critic Learning in Off-Policy Reinforcement Learning
Apr 21, 2026Yuan Zhuang, Yuexin Bian, Sihong He +7Critic Learning Off-Policy Learning
University of Connecticut · University of California San Diego · University of Texas at Arlington +1

Aug 3, 2026, cs.LGJ/K move · Enter open · S save

Diffusion Policy with Behavioral Advantage Correction for Offline Reinforcement Learning

Botao Dong, Longyang Huang, Ning Pang, Hongtian Chen

Botao Dong is with the School of Chemistry, Chemical Engineering and Biotechnology, Nanyang Technological University, Singapore 637459

Diffusion Policies Offline Reinforcement Learning

Open paper AskPDF arXiv

In offline reinforcement learning (RL), the distribution shift between behavioral data and the learned policy can lead to erroneous \emph{Q}-value estimation, thereby misguiding the direction of policy optimization. To address this issue, we develop a behavioral advantage corrected policy evaluation (BAC-PE) approach, which utilizes the \emph{Q}-function of the behavior policy to correct the learned policy's \emph{Q}-function, thus mitigating pessimistic conservatism and overestimation bias. Furthermore, the convergence of BAC-PE is analyzed theoretically, and an upper bound on the difference between the learned \emph{Q}-function and the true \emph{Q}-function is derived. To alleviate distribution shift, this work employs diffusion models to represent both the behavior policy and the learned policy, performing distribution matching for accurate policy regularization. Additionally, \emph{Q}-value guidance is incorporated into the training process to achieve effective policy improvement. By combining BAC-PE with diffusion policy modeling, we propose the diffusion policy with behavioral advantage correction (DPBAC) algorithm. Compared to existing offline methods, DPBAC demonstrates stronger policy representation capabilities and effectively mitigates the bias in \emph{Q}-value estimation. Experimental results on multiple domains of D4RL tasks show that DPBAC achieves superior performance, with notable advantages over state-of-the-art (SOTA) algorithms.

Abstract

Explore similar work

Diffusion Policy with Behavioral Advantage Correction for Offline Reinforcement Learning

Ratio-Variance Regularized Policy Optimization

Low-Rank Adaptation for Critic Learning in Off-Policy Reinforcement Learning