Decoupling Policy Extraction for Offline Reinforcement Learning

Xuyao Lin; Yixiang Shan; Jinru Duan; Tao Yang; Xinyu Zhao; Runyu Lei; Yiming Zhao; Jiaxin Fan

cs.LGAug 21, 2026

Decoupling Policy Extraction for Offline Reinforcement Learning

Authors: Xuyao Lin, Yixiang Shan, Jinru Duan, Tao Yang, Xinyu Zhao, Runyu Lei, Yiming Zhao, Jiaxin Fan, +2 more

Organizations: Simplexity Robotics, Rensselaer Polytechnic Institute · Simplexity Robotics · Simplexity Robotics, Northeastern University

Abstract

Offline RL methods commonly jointly train the actor and critic, where the critic is used to guide the actor toward higher-value actions. This coupled learning process is well motivated in online RL, where an improved actor collects new data that can further update the actor and the critic. However, training data remains fixed in offline RL, making actor-side policy improvement unable to generate new data to validate or correct the critic. Moreover, retaining this coupled paradigm leads to two related challenges. Firstly, actor updates can drift toward high-valued but potentially out-of-distribution (OOD) actions and amplify critic overestimation. Secondly, conservative value estimation or behavior-cloning regularization creates a difficult trade-off between suppressing OOD actions and selecting high-value actions within the data-supported region. Motivated by this observation, we revisit the conventional offline RL paradigm and propose decoupling policy improvement from actor training. Specifically, we train the actor solely to model the behavior distribution and perform policy improvement at inference time by reranking multiple actor-generated proposals with a separately learned critic. We refer to this paradigm as the decoupled policy extraction paradigm. Under such paradigm, the actor provides behavior-supported action candidates, while the critic performs value-based selection within this candidate set. Extensive experiments show that the decoupled policy extraction paradigm outperforms both behavior cloning and jointly learned offline RL methods, while remaining effective even with a naive Q-learning critic.

Figures & tables

Figure 1: Aggregate performance on 30 offline GCRL tasks. Decoupled policy extraction variants outperforms original coupled baselines. Full task-level results and hyperparameters are provided in Section A.3 .

Figure 2: Dashed contours mark the same-checkpoint 95% conformal BC reference regions, with outside rates reported in each panel. Colors encode raw diagnostic advantages for CRR and within-checkpoint $\min(Q_{1},Q_{2})$ percentile ranks for TD3.

Figure 3: Observed success and BC support across selected CQL conservatism settings on Cube Double. The coefficient that maximizes success differs from that minimizing Outside BC95, while stronger pessimism eventually reduces success without improving the support diagnostic.

Dataset	Task	FM-BC	FM-IQL	TRL	BC + critic reranking
					IQL	Transitive	Q-Learning
AntMaze-Large	task1	22 $\pm$ 10	32 $\pm$ 4	52 $\pm$ 8	59 $\pm$ 17	73 $\pm$ 7	28 $\pm$ 22
	task2	24 $\pm$ 3	29 $\pm$ 7	52 $\pm$ 12	11 $\pm$ 3	27 $\pm$ 11	23 $\pm$ 7
	task3	49 $\pm$ 10	73 $\pm$ 9	81 $\pm$ 9	81 $\pm$ 7	85 $\pm$ 8	67 $\pm$ 9
	task4	12 $\pm$ 9	10 $\pm$ 4	19 $\pm$ 2	5 $\pm$ 5	19 $\pm$ 19	23 $\pm$ 9
	task5	17 $\pm$ 3	12 $\pm$ 6	27 $\pm$ 7	15 $\pm$ 8	39 $\pm$ 11	55 $\pm$ 13

Table 1: OGBench success rates (%; mean

\pm

s.d. over five seeds when reported).

FM-IQL Variants	N = 1	N = 2	N = 4	N = 8	N = 16	N = 32
Coupled (Original)	32.0±3.9	N/A	N/A	N/A	N/A	N/A
Coupled + Reranking	32.0±4.7	63.6±2.7	71.6±10.6	64.9±2.3	68.9±8.3	66.2±6.7
Decoupled + Reranking (Ours)	0.0±0.0	32.0±2.2	64.4±6.7	75.6±6.2	78.2±7.7	76.4±4.1

Table 2: Overall performance of FM-IQL variants on Puzzle-4x4 across different candidate budgets

N

Figure 4: Impact of the number of candidates.

Method	$N$	Success (%)	Latency (ms)	Latency ratio	GFLOPs	GPU mem. (MiB)
FM-BC	1	$7.20\pm 2.00$	0.31	1.00 $\times$	0.011	348
FM-BC	32 (random)	$6.67\pm 1.46$	0.40	1.29 $\times$	0.362	542
IQL-Q	1	$5.33\pm 1.74$	0.33	1.09 $\times$	0.011	356
	2	$69.33\pm 1.12$	0.50	1.63 $\times$	0.026	550
	4	$\mathbf{77.07\pm 1.86}$	0.51	1.68 $\times$	0.052	550
	8	$69.33\pm 1.33$	0.50	1.64 $\times$	0.105	550

Table 3: Inference cost under different candidate budgets on the Cube-Double environment.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Gradient steps per trained model	$10^{6}$
Optimizer	Adam
Learning rate	$3\times 10^{-4}$
Batch size	$1024$
Actor hidden dimensions	$[512,512,512]$
Critic/value hidden dimensions	$[512,512,512]$

Appendix

Table 4: Hyperparameters shared across the standard OGBench experiments.

Environment	$\gamma$	$\alpha_{\mathrm{FM}}$	$\lambda_{\mathrm{tr}}$	BC $+$ IQL-Q $N_{1:5}$	BC $+$ Transitive-Q $N_{1:5}$
AntMaze-Large	$0.99$	$1.0$	$0.0$	$(16,1,2,1,1)$	$(32,1,2,2,2)$
HumanoidMaze-Med.	$0.995$	$0.4$	$0.0$	$(16,8,2,1,2)$	$(16,8,4,4,8)$
AntSoccer-Arena	$0.99$	$0.4$	$0.5$	$(8,8,4,8,8)$	$(16,8,8,16,4)$
Cube-Double	$0.99$	$2.5$	$1.0$	$(4,4,4,16,4)$	$(4,4,4,4,2)$
Scene-Play	$0.99$	$2.5$	$1.0$	$(2,8,4,8,2)$	$(8,8,8,16,16)$
Puzzle-4x4	$0.99$	$2.5$	$2.0$	$(4,16,32,32,16)$	$(8,8,32,16,32)$

Appendix

Table 5: Environment- and task-specific hyperparameters. The vectors contain

(N_{1},\ldots,N_{5})

in the task order used by the corresponding OGBench environment. For BC

+

IQL-Q and BC

+

Transitive-Q,

N

is selected independently for every method and task from

\{1,2,4,8,16,32\}

using the five-seed mean.

Environment	Subtask	CRL	QRL
AntMaze-Large	Task 1	$87.56\pm 3.28$	$56.44\pm 6.49$
	Task 2	$87.11\pm 3.50$	$69.11\pm 4.63$
	Task 3	$87.78\pm 3.71$	$92.44\pm 2.27$
	Task 4	$88.00\pm 2.49$	$55.33\pm 8.02$
	Task 5	$89.78\pm 1.37$	$61.56\pm 5.48$
	overall	$88.04\pm 1.49$	$66.98\pm 3.46$

Appendix

Table 6: CRL and QRL success rates. Each seed-level value first averages the 800K, 900K, and 1M checkpoints.

Environment	$\gamma$	CRL $\alpha_{\mathrm{BC}}$	QRL $\alpha_{\mathrm{BC}}$
AntMaze-Large	$0.99$	$0.1$	$0.003$
HumanoidMaze-Medium	$0.995$	$0.1$	$0.001$
AntSoccer-Arena	$0.99$	$0.3$	$0.003$
Cube-Double	$0.99$	$3.0$	$0.3$
Scene-Play	$0.99$	$3.0$	$0.3$
Puzzle-4x4	$0.99$	$3.0$	$0.3$

Appendix

Table 7: Environment-specific CRL and QRL hyperparameters.

Hyperparameter	Value
Shared training settings
Environment	antsoccer-arena-navigate-oraclerep-v0
Gradient steps	$10^{6}$ per method
Optimizer	AdamW
Learning rate	$3\times 10^{-4}$
Batch size	$1024$

Appendix

Table 8: Hyperparameters for the diffusion-policy actor-drift diagnostic on AntSoccer-Arena.

Setting	Value
Environment	cube-double-play-oraclerep-v0
Gradient updates	$10^{6}$
Optimizer / learning rate	Adam / $3\times 10^{-4}$
Batch size	$1024$
Actor	$3\times 512$ MLP, LayerNorm, tanh output
Critic	Double $Q$ , $3\times 512$ MLP, LayerNorm

Appendix

Table 9: Shared settings for the CQL conservatism diagnostic.

Explore similar work

Feb 20, 2026

Flow Actor-Critic for Offline Reinforcement Learning

The dataset distributions in offline reinforcement learning (RL) often exhibit complex and multi-modal distributions, necessitating expressive policies to capture such distributions beyond widely-used Gaussian policies. To handle such complex and multi-modal datasets, in this paper, we propose Flow Actor-Critic, a new actor-critic method for offline RL, based on recent flow policies. The proposed method not only uses the flow model for actor as in previous flow policies but also exploits the expressive flow model for conservative critic acquisition to prevent Q-value explosion in out-of-data regions. To this end, we propose a new form of critic regularizer based on the flow behavior proxy model obtained as a byproduct of flow-based actor design. Leveraging the flow model in this joint way, we achieve new state-of-the-art performance for test datasets of offline RL including the D4RL and recent OGBench benchmarks.

Jongseong Chae, Jongeui Park, Yongjae Shin +3

KAIST · UNIST

Reinforcement Learning Offline RL Actor-Critic Methods

Ask PDF

Aug 27, 2026

Simple Actors and Deep Critics for Scalable Reinforcement Learning

Recent progress in offline reinforcement learning (RL) has been driven by expressive generative actors such as diffusion and flow-matching policies, which capture multimodal behavior in offline datasets. However, these actors require multiple denoising or integration steps per action and thus incur substantial overhead at every decision in deployment. In this work, we revisit where capacity should be invested in an offline actor--critic method. Since the critic is used only during training and is discarded at deployment while the actor runs at every decision step, allocating capacity to the critic rather than the actor is more favorable for inference-time efficiency. However, scaling MLP critics in offline RL is known to introduce several distinct instabilities that have, in practice, kept critics shallow. We identify three distinct failure modes that arise when critics are deepened in offline RL---optimization, bootstrap-noise amplification, and value-range drift---and address each with a corresponding ingredient: a residual MLP backbone, n-step bootstrap targets, and a categorical cross-entropy loss. Combining these ingredients with a lightweight deterministic actor, we propose LAC (Light Actor, deep Critic). On OGBench, LAC matches the strongest diffusion- and flow-matching baselines while achieving up to 4x lower inference latency, comparable to one-step distilled policies without distillation. Its critic recipe also transfers across actor parametrizations.

Guhyeon Kang, Jaehwi Lee, Minhae Kwon

Sungkyunkwan University Republic of Korea · Department of Electrical and Computer Engineering, Sungkyunkwan University, Republic of Korea · Soongsil University Republic of Korea

Offline RL Actor-Critic Methods

Ask PDF

Feb 27, 2026

Beyond State-Wise Mirror Descent: Offline Policy Optimization with Parametric Policies

We investigate the theoretical aspects of offline reinforcement learning (RL) under general function approximation. While prior works (e.g., Xie et al., 2021) have established the theoretical foundations of learning a good policy from offline data via pessimism, existing algorithms that are computationally tractable (often in an oracle-efficient sense), such as PSPI, only apply to finite and small action spaces. Moreover, these algorithms rely on state-wise mirror descent and require actors to be implicitly induced from the critic functions, failing to accommodate standalone policy parameterization which is ubiquitous in practice. In this work, we address these limitations and extend the theoretical guarantees to parameterized policy classes over large or continuous action spaces. When extending mirror descent to parameterized policies, we identify contextual coupling as the core difficulty, and show how connecting mirror descent to natural policy gradient leads to novel analyses, guarantees, and algorithmic insights, including a surprising unification between offline RL and imitation learning.

Xiang Li, Yuheng Zhang, Nan Jiang

Nanjing University · UIUC

Offline RL Policy Gradient

Ask PDF