Positive-Only Drifting Policy Optimization

Qi Zhang

cs.LGApr 15, 2026

Positive-Only Drifting Policy Optimization

Authors: Qi Zhang

Abstract

In the field of online reinforcement learning (RL), traditional Gaussian policies and flow-based methods are often constrained by their unimodal expressiveness, complex gradient clipping, or stringent trust-region requirements. Moreover, they all rely on post-hoc penalization of negative samples to correct erroneous actions. This paper introduces Positive-Only Drifting Policy Optimization (PODPO), a likelihood-free and gradient-clipping-free generative approach for online RL. By leveraging the drifting model, PODPO performs policy updates via advantage-weighted local contrastive drifting. Relying solely on positive-advantage samples, it elegantly steers actions toward high-return regions while exploiting the inherent local smoothness of the generative model to enable proactive error prevention. In doing so, PODPO opens a promising new pathway for generative policy learning in online settings.

Figures & tables

Figure 1: Comparison of PODPO and PPO on the Unitree GO2 quadruped gait locomotion task in the Genesis simulator [ 8 ] , using identical network architecture and training configurations. The experimental results show that PODPO achieves approximately 6.7% higher converged return than PPO, validating the superior expressiveness and exploration capability of multimodal policies in high-dimensional continuous action spaces. Although PODPO exhibits mild oscillations during the early training phase—arising from its broad exploration of the multimodal action space—once the optimal gait pattern is discovered, the policy rapidly stabilizes and consistently maintains high performance.

Figure 2: In the Genesis simulator [ 8 ] , we evaluate a challenging 448-step high-difficulty dance motion tracking task under the demanding condition of no historical observations in the input (obs). PPO’s unimodal Gaussian policy is highly sensitive to control frequency and tends to collapse due to its limited single-mode action distribution. In contrast, PODPO, powered by its multimodal generative policy, adapts significantly better to dynamic environments and varying control frequencies.

Figure 3: Ablation study on advantage weighting conducted in the Genesis simulator for the high-difficulty dance motion tracking task. The curves compare training with $\beta|\hat{A}|$ weighting on positive samples versus training without any advantage weighting. The results clearly demonstrate that applying advantage weighting yields substantially better performance. This confirms that assigning higher weights to higher-advantage (i.e., better) samples in the early training phase effectively accelerates convergence. We adopt a simple linear weighting with $\beta=0.1$ ; alternative nonlinear schemes such as $\ln(1+x)$ or $\tanh(x)$ could also be explored. We additionally experimented with scaling the norm $\|V\|$ . Note that, in gradient-based optimization, scaling the magnitude of $V$ during the update is mathematically equivalent to weighting the loss term.

Figure 4: In the Genesis simulator, we train the high-difficulty dance motion tracking task under two different numbers of parallel environments (16384 versus 4096). The results clearly demonstrate that a higher proportion of repeated actions among the positive samples (relative to the total number of actions) leads to superior performance. This finding is consistent with PPO. However, PODPO exhibits a stronger dependence on a large number of parallel environments.

Figure 5: In the Genesis simulator, we compare single-temperature versus multi-temperature configurations on the high-difficulty dance motion tracking task. The results clearly demonstrate that the multi-temperature design is highly beneficial for training.

t	0.02	0.15	2.0
ESS_ratio	0.1123	0.4798	0.8612
Max_p	0.8833	0.6711	0.1354

Table 1: Multi-Temperature ESS Metrics

Figure 6: Ablation study on the number of candidate actions $G$ for the high-difficulty dance motion tracking task in the Genesis simulator. Training curves for $G=4$ , $G=8$ (default), and $G=16$ . When $G=4$ , the contrastive set per observation contains too few negative candidates, leading to highly unstable drifting vector estimation. The policy is easily pulled off track and collapses catastrophically to near-zero reward. $G=16$ achieves strong final performance but trains slower than $G=8$ . In the same wall-clock time, increasing the number of parallel environments yields better overall results than simply increasing $G$ to 16. Thus, we recommend $G=8$ as the default choice.

Policy Optimization

Policy optimization develops algorithms for improving decision rules in reinforcement learning, language models, and sequential decision systems.

See all 221 papers

Drift, Then Spread: Learning One-Step Generative Policies for Online Reinforcement Learning
Oct 11, 2026Chunyu Luo, Hongyu Zhao, Guangyi Ni +2Reinforcement Learning Maximum Entropy RL
The University of Hong Kong · Beihang University · Harbin Institute of Technology, Shenzhen +2
GenPO++: Generative Policy Optimization with Jacobian-free Likelihood Ratios
Jun 5, 2026Ke Hu, Shutong Ding, Panxin Tao +2RL Control Policy Gradient Methods
ShanghaiTech University
CoDrift: Compositional Drifting for Offline Reinforcement Learning
Aug 25, 2026Xiewei Ni, Ruofeng Mei, Xiangyu XuOffline RL Policy Learning
Xi’an Jiaotong University

Oct 11, 2026J/K move · Enter open · S save

Drift, Then Spread: Learning One-Step Generative Policies for Online Reinforcement Learning

Chunyu Luo, Hongyu Zhao, Guangyi Ni, Yue Song+1

The University of Hong Kong · Beihang University · Harbin Institute of Technology, Shenzhen+2

Reinforcement Learning Maximum Entropy RL

Open paper AskPDF arXiv

Generative policies offer expressive action distributions for online reinforcement learning (RL), but many diffusion- and flow-based policies require costly iterative sampling. We propose DriftSpread, a one-step generative policy based on Drifting for online maximum-entropy (MaxEnt) RL. To incorporate entropy without a tractable policy density, we build on an energy-based view of Drifting and derive critic-guided Drift and entropy-induced Spread from the negative MaxEnt objective. Our method learns these updates sequentially in the same actor, using action regression for value improvement and distribution matching to capture the effect of entropy. The actor updates require neither direct samples from the critic-defined Boltzmann target nor policy-density or score estimation. Under exact fitting with a fixed critic, we prove convergence to the Boltzmann target in the small-step limit followed by the long-time limit. On MuJoCo, DriftSpread achieves the highest aggregate normalized return among all evaluated methods, while outperforming the diffusion- and flow-based baselines in both training and inference efficiency.

Abstract

Policy Optimization

Explore similar work

Drift, Then Spread: Learning One-Step Generative Policies for Online Reinforcement Learning

GenPO++: Generative Policy Optimization with Jacobian-free Likelihood Ratios

CoDrift: Compositional Drifting for Offline Reinforcement Learning