Value-Guided Repulsion for Offline Reinforcement Learning

Yixue Hao; Jiahao Zhang; Jiesong Lian; Xufei Guo; Xumeng Guo; Wenhui Zhang; Long Hu; Rui Wang

cs.AIOct 14, 2026

Value-Guided Repulsion for Offline Reinforcement Learning

Authors: Yixue Hao, Jiahao Zhang, Jiesong Lian, Xufei Guo, Xumeng Guo, Wenhui Zhang, Long Hu, Rui Wang

Organizations: School of Computer Science and Technology, Huazhong University of Science and Technology

Abstract

Drifting models have emerged as a highly efficient and stable paradigm for policy design in offline reinforcement learning (RL). However, standard drift policies determine repulsion solely from geometric relationships among generated actions and therefore cannot distinguish them by expected value. To address this limitation, we introduce Value-Guided Repulsion Q-Learning (VRQL) to integrate action value estimates into the particle interaction field. For each state, VRQL combines geometric kernel distance with a normalized Q value scaling. This mechanism assigns greater repulsive influence to particles with lower values while strictly retaining attraction toward behavior action. Consequently, this value-guided interaction field aligns behavior regularization with policy improvement without adding an auxiliary behavior model stage. On 50 OGBench task configurations and 6 D4RL AntMaze configurations, VRQL demonstrates highly competitive performance against established baselines, suggesting that value-guided particle interaction is an effective method for offline RL.

Figures & tables

Figure 1: (Left) Standard drift policies rely on geometric distance, assigning identical repulsive weights ( $W_{A}=W_{B}$ ) to equidistant neighbors particle A and B. (Right) VRQL incorporates critic estimates into the interaction field. The lower valued neighbor A exerts a stronger repulsive force ( $W_{A}>W_{B}$ ), pushing the generated particle $x_{0}$ toward the higher value region at $x^{*}$ . Background indicate the Q value landscape.

OGBench
	Gaussian	Diffusion	Flow	Value-Guide	Drift
Task category	IQL	IDQL	FQL	Q-Flow	DriftQL	VRQL
antmaze-large (5 tasks)	$53\pm 3$	$21\pm 5$	$79\pm 3$	$88\pm 5$	$92\pm 4$	$\textbf{97}\pm 1$
antmaze-giant (5 tasks)	$4\pm 1$	$0\pm 0$	$9\pm 6$	$40\pm 6$	$60\pm 2$	$\textbf{74}\pm 4$
humanoidmaze-medium (5 tasks)	$33\pm 2$	$1\pm 0$	$58\pm 5$	$83\pm 4$	$62\pm 2$	$\textbf{88}\pm 8$
humanoidmaze-large (5 tasks)	$2\pm 1$	$1\pm 0$	$4\pm 2$	$\textbf{8}\pm 2$	$5\pm 8$	$\textbf{8}\pm 1$

Table 1: Offline reinforcement learning results. VRQL achieves the best or tied for best performance across the OGBench categories and the D4RL AntMaze benchmark. The reported

\pm

values aggregate the seed standard deviations across individual tasks. This table presents one representative method from each policy class, while Appendix C provides complete results for all fourteen methods. Boldface indicates the best score in each row.

Panel of Figure 2 — Figure 2: Effect of repulsion weight assignment. The learned assignment used by VRQL is compared with geometric repulsion ( none ), shuffled value biases ( permuted ), and reversed value biases ( inverse ). Curves show mean success rates, and shaded regions denote one standard deviation.

Figure 3: Value normalization ablation. Success rates under different normalization strategies.

Panel of Figure 4 — Figure 4: Sensitivity to the guidance temperature $\beta$ . Training curves for varying $\beta$ on three OGBench tasks. Curves show mean success rates, and shaded regions denote one standard deviation.

Panel of Figure 5 — Figure 5: Sensitivity to the particle count $N$ . Normalized return on three D4RL AntMaze tasks evaluating sensitivity to the generated particle count $N$ . Curves show mean normalized returns, and shaded regions denote one standard deviation.

Figure 6: Relative training time. VRQL training time scales with particle count.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Symbol	Value
General training settings
Optimizer	–	Adam
Actor / critic learning rate	–	$3\times 10^{-4}$
Batch size	–	$256$
Hidden dimensions (actor and critic)	–	$(512,512,512,512)$
Activation	–	GELU

Appendix

Table 2: Shared hyperparameters. Symbols refer to the main text where one is defined.

Hyperparameter	Default	Alternatives
Value scale $S_{\mathrm{norm}}$	$S_{\mathrm{std}}$ (Eq. 10 )	$S_{\mathrm{abs}}$ , $S_{\mathrm{none}}$ (Eq. 15 )
Ensemble aggregation $\operatorname{Agg}$	$\min$	mean
Critic for the value bias	target	online
Value scale stabilizer $\epsilon$	$10^{-6}$	–

Appendix

Table 3: Value-guided repulsion settings. These are the defaults of Section 3.3 ; the ablations of Section 4.3 vary them one at a time.

	Gaussian			Diffusion			Flow
Task	BC	IQL	ReBRAC	IDQL	SRPO	CAC	FAWAC
antmaze-large-navigate-singletask
task1-v0 $\star$	$0\pm 0$	$48\pm 9$	$91\pm 10$	$0\pm 0$	$0\pm 0$	$42\pm 7$	$1\pm 1$
task2-v0	$6\pm 3$	$42\pm 6$	$88\pm 4$	$14\pm 8$	$4\pm 4$	$1\pm 1$	$0\pm 1$
task3-v0	$29\pm 5$	$72\pm 7$	$51\pm 18$	$26\pm 8$	$3\pm 2$	$49\pm 10$	$12\pm 4$
task4-v0	$8\pm 3$	$51\pm 9$	$84\pm 7$	$62\pm 25$	$45\pm 19$	$17\pm 6$	$10\pm 3$

Appendix

Table 4: Full results across all evaluated OGBench and D4RL tasks, methods one to seven, Boldface marks the best of all fourteen methods on a task, a row’s best may appear in Table 5 , which gives the remaining seven methods.

	Flow			Value-Guide		Drift
Task	FBRAC	IFQL	FQL	Q-Flow	GFP	DriftQL	VRQL
antmaze-large-navigate-singletask
task1-v0 $\star$	$70\pm 20$	$24\pm 17$	$80\pm 8$	$95\pm 4$	$95\pm 0$	$95\pm 2$	$\textbf{99}\pm 1$
task2-v0	$35\pm 12$	$8\pm 3$	$57\pm 10$	$85\pm 6$	$\textbf{92}\pm 3$	$85\pm 9$	$91\pm 0$
task3-v0	$83\pm 15$	$52\pm 17$	$93\pm 3$	$93\pm 4$	$95\pm 2$	$97\pm 1$	$\textbf{99}\pm 1$
task4-v0	$37\pm 18$	$18\pm 8$	$80\pm 4$	$79\pm 23$	$90\pm 2$	$91\pm 1$	$\textbf{97}\pm 2$

Appendix

Table 5: Full results across all evaluated OGBench and D4RL tasks, methods eight to fourteen, continuing Table 4 under the same conventions; boldface again marks the best of all fourteen methods on a task.

Offline RL

Offline reinforcement learning learns decision-making policies from fixed datasets without interacting with the environment during training.

See all 297 papers

Decoupling Policy Extraction for Offline Reinforcement Learning
Aug 21, 2026Xuyao Lin, Yixiang Shan, Jinru Duan +7Offline RL Policy Learning
Simplexity Robotics, Rensselaer Polytechnic Institute · Simplexity Robotics · Simplexity Robotics, Northeastern University
Drift Q-Learning
May 29, 2026Anas Houssaini, Mohamad H. Danesh, Amin Abyaneh +3Reinforcement Learning Q-Learning
McGill University · Mila - Quebec AI Institute
Peng's Q( $λ$ ) for Conservative Value Estimation in Offline Reinforcement Learning
May 14, 2026Byeongchan Kim, Min-hwan OhReinforcement Learning Value Function Estimation
Seoul National University

Aug 21, 2026J/K move · Enter open · S save

Decoupling Policy Extraction for Offline Reinforcement Learning

Xuyao Lin, Yixiang Shan, Jinru Duan, Tao Yang+6

Simplexity Robotics, Rensselaer Polytechnic Institute · Simplexity Robotics · Simplexity Robotics, Northeastern University

Offline RL Policy Learning

Open paper AskPDF arXiv

Offline RL methods commonly jointly train the actor and critic, where the critic is used to guide the actor toward higher-value actions. This coupled learning process is well motivated in online RL, where an improved actor collects new data that can further update the actor and the critic. However, training data remains fixed in offline RL, making actor-side policy improvement unable to generate new data to validate or correct the critic. Moreover, retaining this coupled paradigm leads to two related challenges. Firstly, actor updates can drift toward high-valued but potentially out-of-distribution (OOD) actions and amplify critic overestimation. Secondly, conservative value estimation or behavior-cloning regularization creates a difficult trade-off between suppressing OOD actions and selecting high-value actions within the data-supported region. Motivated by this observation, we revisit the conventional offline RL paradigm and propose decoupling policy improvement from actor training. Specifically, we train the actor solely to model the behavior distribution and perform policy improvement at inference time by reranking multiple actor-generated proposals with a separately learned critic. We refer to this paradigm as the decoupled policy extraction paradigm. Under such paradigm, the actor provides behavior-supported action candidates, while the critic performs value-based selection within this candidate set. Extensive experiments show that the decoupled policy extraction paradigm outperforms both behavior cloning and jointly learned offline RL methods, while remaining effective even with a naive Q-learning critic.

Abstract

Offline RL

Explore similar work

Decoupling Policy Extraction for Offline Reinforcement Learning

Drift Q-Learning

Peng's Q(λλλ) for Conservative Value Estimation in Offline Reinforcement Learning

Peng's Q( $λ$ ) for Conservative Value Estimation in Offline Reinforcement Learning