Organizations: The University of Hong Kong · Beihang University · Harbin Institute of Technology, Shenzhen · Independent Researcher · Tsinghua University
Generative policies offer expressive action distributions for online reinforcement learning (RL), but many diffusion- and flow-based policies require costly iterative sampling. We propose DriftSpread, a one-step generative policy based on Drifting for online maximum-entropy (MaxEnt) RL. To incorporate entropy without a tractable policy density, we build on an energy-based view of Drifting and derive critic-guided Drift and entropy-induced Spread from the negative MaxEnt objective. Our method learns these updates sequentially in the same actor, using action regression for value improvement and distribution matching to capture the effect of entropy. The actor updates require neither direct samples from the critic-defined Boltzmann target nor policy-density or score estimation. Under exact fitting with a fixed critic, we prove convergence to the Boltzmann target in the small-step limit followed by the long-time limit. On MuJoCo, DriftSpread achieves the highest aggregate normalized return among all evaluated methods, while outperforming the diffusion- and flow-based baselines in both training and inference efficiency.
Figures & tables
Figure 1: Normalized return versus average training time (a), and average inference time per action (b), across five MuJoCo tasks.
Figure 2: Overview of DriftSpread . (a) Training alternates between critic learning and sequential Drift–Spread actor updates, using action regression for Drift and distribution matching for Spread. The maps fθ represent the same actor at different training stages, each generating an action sample from reference noise in one forward pass. (b) The surface schematically represents the negative MaxEnt objective over policy distributions. The samples below illustrate how Drift moves actions toward higher critic values and Spread then broadens the post-Drift distribution to capture the effect of entropy.
Figure 3: Sequential versus joint fitting for a Gaussian Boltzmann target. Left: target samples. Right: samples from the fitted policies. Joint’s loss weight is tuned to minimize KL divergence to the known target.
Figure 4: Learning a ring-shaped Boltzmann distribution.
Figure 5: Comparison with 13 baselines on five MuJoCo-v3 locomotion tasks. Curves show five-seed mean evaluation returns with sample standard deviations shaded and no smoothing.
Figure 6: Effect of spreading noise on MuJoCo. The four noise choices use the same actor learning rates within each task.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Noise
Characteristic function mη(ω)
Noise sample ϵη
Gaussian
e−c∥ω∥2
2cξ
Resolvent
(1+c∥ω∥2)−1
2cv1ξ
Gamma mixture
(1+c∥ω∥2/κ)−κ
2cvκξ
Uniform
∏j=1dsinc(aηωj)
aηU
Appendix
Table 1: Spreading noise and characteristic functions, with c=λη and aη=6c . The Gamma construction scales a Gaussian vector by an independent Gamma-distributed variance factor.
Figure 8: Sequential and joint fitting of the Gaussian target. (a) Boltzmann target. (b,c) Samples from the fitted actors. Scatter plots use normalized coordinates ai/σi∗ , so the target has unit variance in both coordinates. (d) Final KL divergence for different step sizes at the same evolution time τ=40 . Joint uses the selected weight β≈17.2624 .
Figure 9: Learning the ring-shaped target at different temperatures. Columns vary the entropy coefficient λ . Rows show target samples, Drift-only samples, and samples from DriftSpread . At λ=0 , the reference is uniform on the circle and both training procedures omit Spread.
Figure 10: Sequential and joint fitting on Humanoid-v3. Joint weights the Spread MMD loss by β , with β=0 using the Drift loss alone. Sequential is labeled Two-stage in the figure. Curves show unsmoothed means over five seeds, with sample standard deviations shaded.
Figure 11: Effect of the frequency scale on Humanoid-v3. We multiply the default frequency standard deviation by 0.1 , 1 , or 10 , keeping the spreading noise and other training settings fixed. Curves show unsmoothed means over five seeds, with sample standard deviations shaded.
Method
Repository
MFPO
dongxiaoyi-xyz/MFPO
DIME
ALRhub/DIME
FlowRL
bytedance/FlowRL
SAC-Flow
Elessar123/SAC-FLOW
MaxEntDP
diffusionyes/MaxEntDP
DACER
happy-yan/DACER-Diffusion-with-Online-RL
Appendix
Table 2: Source repositories for the baseline implementations. The SAC entry is the implementation referenced by MFPO. DPMD and SDAC share the same repository.
Hyperparameter
MFPO
DIME
FlowRL
SAC-Flow
MaxEntDP
DACER
QVPO
Target smoothing τ
0.005
1
0.95
1
0.005
0.005
0.005
Hidden layers
3
3
3
N/A
3
3
3
Hidden nodes per layer
256
256
256
N/A
256
256
256
Actor learning rate
3×10−4
3×10−4
3×10−4
3×10−4
3×10−4
10−4
3×10−4
Critic learning rate
3×10−4
3×10−4
3×10−4
10−3
3×10−4
10−4
3×10−4
Activation
GELU
GELU
ELU
GELU
Mish
Mish
Mish
Appendix
Table 3: Hyperparameters of the baseline methods on MuJoCo. All methods use a batch size of 256, a discount factor γ=0.99 , and an update-to-data ratio of 1.
Parameter
Value
Parameter
Value
Batch size †
256
Drift samples per state ND
4
Hidden layers †
3
Spread samples per state NCF
32
Hidden width †
256
Frequencies per cycle Mω
64
Update-to-data ratio †
1
Spreading noise
Gaussian
Discount γ
0.99
Entropy coefficient λ
0.04
Target update rate
0.005
Step size η
0.1
Appendix
Table 4: Training settings for DriftSpread on MuJoCo. Entries marked † are aligned across all methods.
Task
Spread learning rate
HalfCheetah-v3
10−5
Hopper-v3
10−5
Walker2d-v3
2×10−5
Ant-v3
1.5×10−5
Humanoid-v3
10−5
Appendix
Table 5: Spread learning rates for DriftSpread . The same task-specific rates are used in the spreading-noise comparison.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
In the field of online reinforcement learning (RL), traditional Gaussian policies and flow-based methods are often constrained by their unimodal expressiveness, complex gradient clipping, or stringent trust-region requirements. Moreover, they all rely on post-hoc penalization of negative samples to correct erroneous actions. This paper introduces Positive-Only Drifting Policy Optimization (PODPO), a likelihood-free and gradient-clipping-free generative approach for online RL. By leveraging the drifting model, PODPO performs policy updates via advantage-weighted local contrastive drifting. Relying solely on positive-advantage samples, it elegantly steers actions toward high-return regions while exploiting the inherent local smoothness of the generative model to enable proactive error prevention. In doing so, PODPO opens a promising new pathway for generative policy learning in online settings.
Many reinforcement learning (RL) tasks have discrete action spaces, but most generative policy methods based on diffusion and flow matching are designed for continuous control. Meanwhile, generative policies usually rely heavily on offline datasets and offline-to online RL is itself challenging, as the policy must improve from new interaction without losing useful behavior learned from static data. To address those challenges, we introduce DRIFT, an online fine-tuning method that updates an offline pretrained continuous time Markov chain (CTMC) policy with an advantage-weighted discrete flow matching loss. To preserve useful pretrained knowledge, we add a path-space penalty that regularizes the full CTMC trajectory distribution, rather than only the final action distribution. For large discrete action spaces, we introduce a candidate-set approximation that updates the actor over a small subset of actions sampled from reference-policy rollouts and uniform exploration. Our theoretical analysis shows that focusing on a small set of high probability actions can reduce computation while keeping the approximation error small, making our method practical for large action spaces. We evaluate on macro-action MinAtar games where offline-pretrained DRIFT outperforms its online-only counterpart across all five games and achieves the highest mean return on three and on oracle free Jericho, DRIFT achieves a mean normalized score of 7.3% with a lightweight GRU encoder, outperforming the evaluated pretrained language-model baselines. Ablations demonstrate the importance of appropriately weighted path-space regularization and reference-guided candidate construction.
Fairoz Nower Khan, Nabuat Zaman Nahim, Peizhong Ju
Department of Computer Science, University of Kentucky
Maximum entropy reinforcement learning (MaxEnt RL) has become a standard framework for sequential decision making, yet its standard Gaussian policy parameterization is inherently unimodal, limiting its ability to model complex multimodal action distributions. This limitation has motivated increasing interest in generative policies based on diffusion and flow matching as more expressive alternatives. However, incorporating such policies into MaxEnt RL is challenging for two main reasons: the likelihood and entropy of continuous-time generative policies are generally intractable, and multi-step sampling introduces both long-horizon backpropagation instability and substantial inference latency. To address these challenges, we propose Truncated Rectified Flow Policy (TRFP), a framework built on a hybrid deterministic-stochastic architecture. This design makes entropy-regularized optimization tractable while supporting stable training and effective one-step sampling through gradient truncation and flow straightening. Empirical results on a toy multigoal environment and 10 MuJoCo benchmarks show that TRFP captures multimodal behavior effectively, outperforms strong baselines on most benchmarks under standard sampling, and remains highly competitive under one-step sampling.