Organizations: The University of Hong Kong · Beihang University · Harbin Institute of Technology, Shenzhen · Independent Researcher · Tsinghua University
Generative policies offer expressive action distributions for online reinforcement learning (RL), but many diffusion- and flow-based policies require costly iterative sampling. We propose DriftSpread, a one-step generative policy based on Drifting for online maximum-entropy (MaxEnt) RL. To incorporate entropy without a tractable policy density, we build on an energy-based view of Drifting and derive critic-guided Drift and entropy-induced Spread from the negative MaxEnt objective. Our method learns these updates sequentially in the same actor, using action regression for value improvement and distribution matching to capture the effect of entropy. The actor updates require neither direct samples from the critic-defined Boltzmann target nor policy-density or score estimation. Under exact fitting with a fixed critic, we prove convergence to the Boltzmann target in the small-step limit followed by the long-time limit. On MuJoCo, DriftSpread achieves the highest aggregate normalized return among all evaluated methods, while outperforming the diffusion- and flow-based baselines in both training and inference efficiency.
Figures & tables
Figure 1: Normalized return versus average training time (a), and average inference time per action (b), across five MuJoCo tasks.
Figure 2: Overview of DriftSpread . (a) Training alternates between critic learning and sequential Drift–Spread actor updates, using action regression for Drift and distribution matching for Spread. The maps fθ represent the same actor at different training stages, each generating an action sample from reference noise in one forward pass. (b) The surface schematically represents the negative MaxEnt objective over policy distributions. The samples below illustrate how Drift moves actions toward higher critic values and Spread then broadens the post-Drift distribution to capture the effect of entropy.
Figure 3: Sequential versus joint fitting for a Gaussian Boltzmann target. Left: target samples. Right: samples from the fitted policies. Joint’s loss weight is tuned to minimize KL divergence to the known target.
Figure 4: Learning a ring-shaped Boltzmann distribution.
Figure 5: Comparison with 13 baselines on five MuJoCo-v3 locomotion tasks. Curves show five-seed mean evaluation returns with sample standard deviations shaded and no smoothing.
Figure 6: Effect of spreading noise on MuJoCo. The four noise choices use the same actor learning rates within each task.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Noise
Characteristic function mη(ω)
Noise sample ϵη
Gaussian
e−c∥ω∥2
2cξ
Resolvent
(1+c∥ω∥2)−1
2cv1ξ
Gamma mixture
(1+c∥ω∥2/κ)−κ
2cvκξ
Uniform
∏j=1dsinc(aηωj)
aηU
Appendix
Table 1: Spreading noise and characteristic functions, with c=λη and aη=6c . The Gamma construction scales a Gaussian vector by an independent Gamma-distributed variance factor.
Figure 8: Sequential and joint fitting of the Gaussian target. (a) Boltzmann target. (b,c) Samples from the fitted actors. Scatter plots use normalized coordinates ai/σi∗ , so the target has unit variance in both coordinates. (d) Final KL divergence for different step sizes at the same evolution time τ=40 . Joint uses the selected weight β≈17.2624 .
Figure 9: Learning the ring-shaped target at different temperatures. Columns vary the entropy coefficient λ . Rows show target samples, Drift-only samples, and samples from DriftSpread . At λ=0 , the reference is uniform on the circle and both training procedures omit Spread.
Figure 10: Sequential and joint fitting on Humanoid-v3. Joint weights the Spread MMD loss by β , with β=0 using the Drift loss alone. Sequential is labeled Two-stage in the figure. Curves show unsmoothed means over five seeds, with sample standard deviations shaded.
Figure 11: Effect of the frequency scale on Humanoid-v3. We multiply the default frequency standard deviation by 0.1 , 1 , or 10 , keeping the spreading noise and other training settings fixed. Curves show unsmoothed means over five seeds, with sample standard deviations shaded.
Method
Repository
MFPO
dongxiaoyi-xyz/MFPO
DIME
ALRhub/DIME
FlowRL
bytedance/FlowRL
SAC-Flow
Elessar123/SAC-FLOW
MaxEntDP
diffusionyes/MaxEntDP
DACER
happy-yan/DACER-Diffusion-with-Online-RL
Appendix
Table 2: Source repositories for the baseline implementations. The SAC entry is the implementation referenced by MFPO. DPMD and SDAC share the same repository.
Hyperparameter
MFPO
DIME
FlowRL
SAC-Flow
MaxEntDP
DACER
QVPO
Target smoothing τ
0.005
1
0.95
1
0.005
0.005
0.005
Hidden layers
3
3
3
N/A
3
3
3
Hidden nodes per layer
256
256
256
N/A
256
256
256
Actor learning rate
3×10−4
3×10−4
3×10−4
3×10−4
3×10−4
10−4
3×10−4
Critic learning rate
3×10−4
3×10−4
3×10−4
10−3
3×10−4
10−4
3×10−4
Activation
GELU
GELU
ELU
GELU
Mish
Mish
Mish
Appendix
Table 3: Hyperparameters of the baseline methods on MuJoCo. All methods use a batch size of 256, a discount factor γ=0.99 , and an update-to-data ratio of 1.
Parameter
Value
Parameter
Value
Batch size †
256
Drift samples per state ND
4
Hidden layers †
3
Spread samples per state NCF
32
Hidden width †
256
Frequencies per cycle Mω
64
Update-to-data ratio †
1
Spreading noise
Gaussian
Discount γ
0.99
Entropy coefficient λ
0.04
Target update rate
0.005
Step size η
0.1
Appendix
Table 4: Training settings for DriftSpread on MuJoCo. Entries marked † are aligned across all methods.
Task
Spread learning rate
HalfCheetah-v3
10−5
Hopper-v3
10−5
Walker2d-v3
2×10−5
Ant-v3
1.5×10−5
Humanoid-v3
10−5
Appendix
Table 5: Spread learning rates for DriftSpread . The same task-specific rates are used in the spreading-noise comparison.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.