ScoRe-Flow: Complete Distributional Control via Score-Based Reinforcement Learning for Flow Matching

Xiaotian Qiu; Lukai Chen; Jinhao Li; Qi Sun; Cheng Zhuo; Guohao Dai

cs.ROApr 13, 2026

ScoRe-Flow: Complete Distributional Control via Score-Based Reinforcement Learning for Flow Matching

Authors: Xiaotian Qiu, Lukai Chen, Jinhao Li, Qi Sun, Cheng Zhuo, Guohao Dai

Organizations: Zhejiang University, Hangzhou, China · Shanghai Innovation Institute · Shanghai Jiao Tong University, Shanghai, China · Infinigence-AI

Abstract

Flow Matching (FM) policies have emerged as an efficient backbone for robotic control, offering fast and expressive action generation that underpins recent large-scale embodied AI systems. However, FM policies trained via imitation learning inherit the limitations of demonstration data; surpassing suboptimal behaviors requires reinforcement learning (RL) fine-tuning. Recent methods convert deterministic flows into stochastic differential equations (SDEs) with learnable noise injection, enabling exploration and tractable likelihoods, but such noise-only control can compromise training efficiency when demonstrations already provide strong priors. We observe that modulating the drift via the score function, i.e., the gradient of log-density, steers exploration toward high-probability regions, improving stability. The score admits a closed-form expression from the velocity field, requiring no auxiliary networks. Based on this, we propose ScoRe-Flow, a score-based RL fine-tuning method that combines drift modulation with learned variance prediction to achieve decoupled control over the mean and variance of stochastic transitions. Experiments demonstrate that ScoRe-Flow achieves 2.4x faster convergence than flow-based SOTA on D4RL locomotion tasks and up to 5.4% higher success rates on Robomimic and Franka Kitchen manipulation tasks.

Figures & tables

Figure 1: Comparison of flow policy sampling strategies. Top : Deterministic FM follows a fixed ODE trajectory with no exploration. Middle : Noise-only control (e.g., ReinFlow) injects learnable noise for exploration, but only perturbs the position without modulating dynamics. Bottom : ScoRe-Flow combines score-based drift modulation with learned variance prediction, achieving decoupled mean-variance control.

Panel of Figure 2 — Figure 2: Learning curves on D4RL locomotion tasks. Dashed lines indicate the behavior cloning level.

Panel of Figure 3 — Figure 3: Learning curves on Robomimic visual manipulation tasks.

Panel of Figure 4 — Figure 4: Learning curves on Franka Kitchen multi-task benchmark.

Figure 5: Wall-clock time comparison on D4RL locomotion tasks. The large speedup over DPPO (up to 94.2 $\times$ on Hopper-v2, 21.9 $\times$ average) is primarily a structural advantage of flow-based methods requiring fewer denoising steps. The algorithmic contribution is the $2.4\times$ faster convergence over the flow-based baseline ReinFlow at matched $K=4$ steps.

D4RL Locomotion (Episode Reward $\uparrow$ )
Task	BC (pretrain)	DPPO	ReinFlow-R	ReinFlow-S	Score-SDE	ScoRe-Flow (Ours)
Hopper-v2	1578 $\pm$ 28	3275 $\pm$ 20	3204 $\pm$ 36	3253 $\pm$ 24	3118 $\pm$ 87	3233 $\pm$ 39
Walker2d-v2	3185 $\pm$ 38	3896 $\pm$ 62	4092 $\pm$ 11	4202 $\pm$ 26	3900 $\pm$ 57	4172 $\pm$ 154
Ant-v0	3750 $\pm$ 118	3951 $\pm$ 24	4021 $\pm$ 25	4119 $\pm$ 44	3980 $\pm$ 29	4088 $\pm$ 95
Humanoid-v3	2415 $\pm$ 65	5001 $\pm$ 20	5041 $\pm$ 6	4801 $\pm$ 134	4896 $\pm$ 91	5100 $\pm$ 47
Avg.	2732	4031	4090	4094	3974	4148

Table 1: Performance comparison across all benchmarks. We report final performance (mean

\pm

std over 3 seeds). The BC (pretrain) column indicates the baseline performance of the behavior cloning policy before fine-tuning. Bold : best, underline : second best.

Figure 6: Training stability comparison on Kitchen-Complete-v0. Under the same Rectified Flow base policy, ScoRe-Flow achieves stable learning while ReinFlow exhibits high variance and unstable convergence behavior.

Figure 7: Sensitivity of Score-SDE to initial noise variance. Performance varies dramatically with different fixed

\sigma

values, demonstrating that score-only methods require careful task-specific tuning.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Optimizer & Learning Rate Settings
Parameter	Value (Common to Both)
Actor Optimizer	Adam ( Kingma, 2014 )
Actor LR Weight Decay	0
Actor LR Scheduler	CosineAnnealingWarmupRestart ( Loshchilov & Hutter, 2016 )
Actor LR Cycle Steps	100 ∗
Critic Optimizer	Adam

Appendix

Table 2: Shared Hyperparameter Configurations for Score-based SDE and ScoRe-Flow.

Table 3: Detailed Hyperparameter Settings for OpenAI Gym Locomotion Tasks.

Table 4: Hyperparameter Configurations for Franka Kitchen Tasks.

Table 5: Hyperparameter Configurations for Robomimic Image-input Manipulation Tasks.

Task	Method	Single Iteration (s)			Mean $\pm$ Std
Task	Method	S1	S2	S3	Mean $\pm$ Std
Hopper-v2	ReinFlow-R	11.598	11.704	11.843	11.715 $\pm$ 0.123
	ReinFlow-S	12.051	12.127	12.372	12.290 $\pm$ 0.141
	DPPO	99.502	99.616	98.021	99.046 $\pm$ 0.890
	FQL	4.373	4.366	4.515	4.418 $\pm$ 0.084
	Score-SDE	11.652	11.715	11.793	11.720 $\pm$ 0.071

Appendix

Table 6: Per Iteration Wall-clock Time Analysis on D4RL OpenAI Gym Locomotion Tasks (Seconds). Comparison of single training iteration time across three random seeds. ScoRe-Flow maintains competitive computational efficiency compared to flow-based baselines, with only negligible overhead introduced by the variance prediction mechanism.

Figure 8: Ablation study on denoising steps (

K

). We evaluate the success rate on the Robomimic Square task across varying inference steps (

K\in\{1,2,4\}

). The results demonstrate that the success rate reaches its maximum at

K=2

, outperforming both the 4-step and 1-step settings, thus identifying

K=2

as the optimal inference configuration.

Domain	Task / Dataset	Model Framework	Rationale
Locomotion	OpenAI Gym (All Tasks)	Rectified Flow	Performance
Manipulation	Franka Kitchen (All Tasks)	Shortcut Model	Performance
	Robomimic Can	Rectified Flow	Performance
	Robomimic Square	Rectified Flow	Performance
	Robomimic Transport	Shortcut Model	Efficiency & Performance

Appendix

Table 7: Flow Matching Model Variants Configuration. We select the specific model architecture (Rectified Flow vs. Shortcut Model) for each task to achieve optimal performance, aligning with the baseline configurations reported in ReinFlow for a fair comparison.

Method	Hopper	Walker2d	Ant	Humanoid
DPPO	3275 $\pm$ 24	3897 $\pm$ 76 †	3951 $\pm$ 30 †	4873 $\pm$ 0
FQL	3032 $\pm$ 65 †	3709 $\pm$ 14 †	620 $\pm$ 3 †	2334 $\pm$ 61 †
ReinFlow-R	3204 $\pm$ 44	4092 $\pm$ 14	4021 $\pm$ 31	4976 $\pm$ 26 †
ReinFlow-S	3253 $\pm$ 29	4202 $\pm$ 32	4119 $\pm$ 53	4956 $\pm$ 110
Score-SDE	3155 $\pm$ 91	3922 $\pm$ 78 †	3975 $\pm$ 51 †	4928 $\pm$ 113 †
ScoRe-Flow	3235 $\pm$ 34	4189 $\pm$ 145	4096 $\pm$ 87	5125 $\pm$ 47

Appendix

Table 8: 5-seed D4RL results (mean

\pm

std).

\dagger

p<0.05

vs ScoRe-Flow (Welch’s t-test).

Method	Can	Square	Transport	K-Complete	K-Mixed	K-Partial
DPPO	.993 $\pm$ .006	.771 $\pm$ .033	.528 $\pm$ .459	3.78 $\pm$ .17	3.40 $\pm$ .54	3.48 $\pm$ .47
ReinFlow-S	.978 $\pm$ .003	.744 $\pm$ .047	.888 $\pm$ .030	3.92 $\pm$ .03	2.99 $\pm$ .01	3.39 $\pm$ .54
Score-SDE	.957 $\pm$ .019	.778 $\pm$ .024 †	.896 $\pm$ .060	3.97 $\pm$ .01	2.99 $\pm$ .01	3.18 $\pm$ .45 †
ScoRe-Flow	.979 $\pm$ .020	.835 $\pm$ .032	.940 $\pm$ .022	3.96 $\pm$ .02	3.00 $\pm$ .01	3.98 $\pm$ .02

Appendix

Table 9: 5-seed Robomimic (success rate) and Kitchen (tasks completed) results.

\dagger

p<0.05

Figure 9: Learned

\alpha_{\psi}(t)

scheduler behavior. Left:

\alpha_{\psi}

as a function of denoising time

t

at different training stages. The scheduler applies stronger correction early in denoising and reduces near

t=1

. Right: Overall

\alpha_{\psi}

magnitude decreases over training as

\mathbf{v}_{\theta}

improves.

Explore similar work

May 22, 2026

Score-Based One-step MeanFlow Policy Optimization

Diffusion and flow matching have emerged as expressive policy classes in reinforcement learning, but their reliance on multi-step denoising imposes substantial computational overhead at inference time, which is particularly problematic in online RL. MeanFlow offers a promising alternative by learning an average velocity field that maps noise to data in a single network evaluation. However, MeanFlow typically requires samples from the target distribution to construct its target velocity field, which are unavailable in online RL. We propose Score-Based One-step MeanFlow Policy Optimization (SOM), an actor-critic algorithm that resolves this by constructing the target velocity field directly from the Q-function via score estimation and a probability flow ODE, thereby concentrating probability mass on high-value modes. In the fully online RL setting, SOM achieves state-of-the-art performance on locomotion tasks with a single generation step, while substantially reducing both training and inference time compared to prior diffusion- and flow-matching-based policies.

Kyungyoon Kim, Donghyeon Ki, Hee-Jun Ahn +1

Korea University, Decision Making Lab · Gauss Labs Inc.

Flow Matching MeanFlow Policy Optimization Mean Flow RL for Diffusion Models

Ask PDF

Oct 12, 2026

A Survey of Flow Matching in Reinforcement Learning

Flow Matching (FM) has recently emerged as a principled and efficient generative modeling framework for reinforcement learning (RL), enabling expressive, multimodal policy parameterizations via deterministic probability transport. Compared to diffusion-based policies that rely on stochastic denoising chains, FM uses sampling based on ordinary differential equations (ODEs), with learned velocity fields, which can substantially reduce inference latency and simplify the incorporation of RL objectives. As research in flow-based RL rapidly accelerates across offline continuous control, online fine-tuning, and foundation model alignment, the literature has become highly fragmented. In this survey, we provide a comprehensive taxonomy of flow-matching approaches in reinforcement learning. We organize the literature along two axes: the target distribution being modeled (e.g., action policies, value critics, transition dynamics) and the mechanism of RL signal integration (e.g., energy-weighted regression, flow-based policy gradients, and group relative policy optimization). Furthermore, we survey emerging frontiers such as discrete and non-Euclidean action spaces, provide a systematic comparative analysis against Gaussian and diffusion baselines, and outline critical open problems. Ultimately, this survey serves as a foundational roadmap for the next generation of generative reinforcement learning and alignment.

Nabuat Zaman Nahim, Fairoz Nower Khan, Peizhong Ju

Department of Computer Science, University of Kentucky, Lexington, Kentucky 40506, USA

Reinforcement Learning Flow-Based Generative Modeling RL for Generative Models

Ask PDF

Feb 2, 2026

Flow Policy Gradients for Robot Control

Likelihood-based policy gradient methods are the dominant approach for training robot control policies from rewards. These methods rely on differentiable action likelihoods, which constrain policy outputs to simple distributions like Gaussians. In this work, we show how flow matching policy gradients, a recent framework that bypasses likelihood computation, can be made effective for training and finetuning more expressive policies in challenging robot control settings. We introduce an improved objective that enables success in legged locomotion, humanoid motion tracking, and manipulation tasks, as well as robust sim-to-real transfer on two humanoid robots. We then present ablations and analysis on training dynamics. Results show how policies can exploit the flow representation for exploration when training from scratch, as well as improved finetuning robustness over baselines.

Brent Yi, Hongsuk Choi, Himanshu Gaurav Singh +9

Amazon FAR · UC Berkeley · Stanford +2

Robotic Control RL for Robotics RL for Legged Locomotion

Ask PDF