Drifting models have emerged as a highly efficient and stable paradigm for policy design in offline reinforcement learning (RL). However, standard drift policies determine repulsion solely from geometric relationships among generated actions and therefore cannot distinguish them by expected value. To address this limitation, we introduce Value-Guided Repulsion Q-Learning (VRQL) to integrate action value estimates into the particle interaction field. For each state, VRQL combines geometric kernel distance with a normalized Q value scaling. This mechanism assigns greater repulsive influence to particles with lower values while strictly retaining attraction toward behavior action. Consequently, this value-guided interaction field aligns behavior regularization with policy improvement without adding an auxiliary behavior model stage. On 50 OGBench task configurations and 6 D4RL AntMaze configurations, VRQL demonstrates highly competitive performance against established baselines, suggesting that value-guided particle interaction is an effective method for offline RL.
Figures & tables
Figure 1: (Left) Standard drift policies rely on geometric distance, assigning identical repulsive weights ( WA=WB ) to equidistant neighbors particle A and B. (Right) VRQL incorporates critic estimates into the interaction field. The lower valued neighbor A exerts a stronger repulsive force ( WA>WB ), pushing the generated particle x0 toward the higher value region at x∗ . Background indicate the Q value landscape.
Gaussian
Diffusion
Flow
Value-Guide
Drift
Task category
IQL
IDQL
FQL
Q-Flow
DriftQL
VRQL
OGBench
antmaze-large (5 tasks)
53±3
21±5
79±3
88±5
92±4
97±1
antmaze-giant (5 tasks)
4±1
0±0
9±6
40±6
60±2
74±4
humanoidmaze-medium (5 tasks)
33±2
1±0
58±5
83±4
62±2
88±8
humanoidmaze-large (5 tasks)
2±1
1±0
4±2
8±2
5±8
8±1
Table 1: Offline reinforcement learning results. VRQL achieves the best or tied for best performance across the OGBench categories and the D4RL AntMaze benchmark. The reported ± values aggregate the seed standard deviations across individual tasks. This table presents one representative method from each policy class, while Appendix C provides complete results for all fourteen methods. Boldface indicates the best score in each row.
Figure 2: Effect of repulsion weight assignment. The learned assignment used by VRQL is compared with geometric repulsion ( none ), shuffled value biases ( permuted ), and reversed value biases ( inverse ). Curves show mean success rates, and shaded regions denote one standard deviation.
Figure 3: Value normalization ablation. Success rates under different normalization strategies.
Figure 4: Sensitivity to the guidance temperature β . Training curves for varying β on three OGBench tasks. Curves show mean success rates, and shaded regions denote one standard deviation.
Figure 5: Sensitivity to the particle count N . Normalized return on three D4RL AntMaze tasks evaluating sensitivity to the generated particle count N . Curves show mean normalized returns, and shaded regions denote one standard deviation.
Figure 6: Relative training time. VRQL training time scales with particle count.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Symbol
Value
General training settings
Optimizer
–
Adam
Actor / critic learning rate
–
3×10−4
Batch size
–
256
Hidden dimensions (actor and critic)
–
(512,512,512,512)
Activation
–
GELU
Appendix
Table 2: Shared hyperparameters. Symbols refer to the main text where one is defined.
Hyperparameter
Default
Alternatives
Value scale Snorm
Sstd (Eq. 10 )
Sabs , Snone (Eq. 15 )
Ensemble aggregation Agg
min
mean
Critic for the value bias
target
online
Value scale stabilizer ϵ
10−6
–
Appendix
Table 3: Value-guided repulsion settings. These are the defaults of Section 3.3 ; the ablations of Section 4.3 vary them one at a time.
Gaussian
Diffusion
Flow
Task
BC
IQL
ReBRAC
IDQL
SRPO
CAC
FAWAC
antmaze-large-navigate-singletask
task1-v0 ⋆
0±0
48±9
91±10
0±0
0±0
42±7
1±1
task2-v0
6±3
42±6
88±4
14±8
4±4
1±1
0±1
task3-v0
29±5
72±7
51±18
26±8
3±2
49±10
12±4
task4-v0
8±3
51±9
84±7
62±25
45±19
17±6
10±3
Appendix
Table 4: Full results across all evaluated OGBench and D4RL tasks, methods one to seven, Boldface marks the best of all fourteen methods on a task, a row’s best may appear in Table 5 , which gives the remaining seven methods.
Flow
Value-Guide
Drift
Task
FBRAC
IFQL
FQL
Q-Flow
GFP
DriftQL
VRQL
antmaze-large-navigate-singletask
task1-v0 ⋆
70±20
24±17
80±8
95±4
95±0
95±2
99±1
task2-v0
35±12
8±3
57±10
85±6
92±3
85±9
91±0
task3-v0
83±15
52±17
93±3
93±4
95±2
97±1
99±1
task4-v0
37±18
18±8
80±4
79±23
90±2
91±1
97±2
Appendix
Table 5: Full results across all evaluated OGBench and D4RL tasks, methods eight to fourteen, continuing Table 4 under the same conventions; boldface again marks the best of all fourteen methods on a task.
Offline RL methods commonly jointly train the actor and critic, where the critic is used to guide the actor toward higher-value actions. This coupled learning process is well motivated in online RL, where an improved actor collects new data that can further update the actor and the critic. However, training data remains fixed in offline RL, making actor-side policy improvement unable to generate new data to validate or correct the critic. Moreover, retaining this coupled paradigm leads to two related challenges. Firstly, actor updates can drift toward high-valued but potentially out-of-distribution (OOD) actions and amplify critic overestimation. Secondly, conservative value estimation or behavior-cloning regularization creates a difficult trade-off between suppressing OOD actions and selecting high-value actions within the data-supported region. Motivated by this observation, we revisit the conventional offline RL paradigm and propose decoupling policy improvement from actor training. Specifically, we train the actor solely to model the behavior distribution and perform policy improvement at inference time by reranking multiple actor-generated proposals with a separately learned critic. We refer to this paradigm as the decoupled policy extraction paradigm. Under such paradigm, the actor provides behavior-supported action candidates, while the critic performs value-based selection within this candidate set. Extensive experiments show that the decoupled policy extraction paradigm outperforms both behavior cloning and jointly learned offline RL methods, while remaining effective even with a naive Q-learning critic.
Xuyao Lin, Yixiang Shan, Jinru Duan +7
Simplexity Robotics, Rensselaer Polytechnic Institute · Simplexity Robotics · Simplexity Robotics, Northeastern University
Offline reinforcement learning requires improving a policy from fixed data while avoiding out-of-distribution actions with unreliable value estimates. Diffusion and flow policies handle this trade-off by modeling the behavior distribution to regularize the RL objective, but they require iterative denoising, solver integrations, and in more efficient variants, distillation or other approximations at inference. We propose DriftQL, which combines a drift-based behavioral regularizer with critic-driven policy improvement. The value signal biases the policy toward high-value regions of the data support, while attraction and repulsion together keep generated actions near the data and prevent collapse onto a single mode. DriftQL is implemented as a single network with a unified training objective and generates actions in a single forward pass. On D4RL and OGBench, DriftQL consistently outperforms diffusion and flow methods, advancing the state of the art. Under degraded data quality, where the baselines visibly struggle, DriftQL remains close to its clean-data performance, positioning it as a promising alternative to diffusion and flow-based methods while maintaining the simplicity and efficiency of deterministic approaches. Project page: https://driftql.github.io/
Anas Houssaini, Mohamad H. Danesh, Amin Abyaneh +3
We propose a model-free offline multi-step reinforcement learning (RL) algorithm, Conservative Peng's Q(λ) (CPQL). Our algorithm adapts the Peng's Q(λ) (PQL) operator for conservative value estimation as an alternative to the Bellman operator. To the best of our knowledge, this is the first work in offline RL to theoretically and empirically demonstrate the effectiveness of conservative value estimation with a multi-step operator by fully leveraging offline trajectories. The fixed point of the PQL operator in offline RL lies closer to the value function of the behavior policy, thereby naturally inducing implicit behavior regularization. CPQL simultaneously mitigates over-pessimistic value estimation, achieves performance greater than (or equal to) that of the behavior policy, and provides near-optimal performance guarantees -- a milestone that previous conservative approaches could not achieve. Extensive numerical experiments on the D4RL benchmark demonstrate that CPQL consistently and significantly outperforms existing offline single-step baselines. In addition to the contributions of CPQL in offline RL, our proposed method also contributes to the offline-to-online learning framework. Using the Q-function pre-trained by CPQL in offline settings enables the online PQL agent to avoid the performance drop typically observed at the start of fine-tuning and to attain robust performance improvements. Our code is available at https://github.com/oh-lab/CPQL.