GoldenStart: Q-Guided Priors and Entropy Control for Distilling Flow Policies

He Zhang; Ying Sun; Hui Xiong

cs.LGMar 15, 2026

GoldenStart: Q-Guided Priors and Entropy Control for Distilling Flow Policies

Authors: He Zhang, Ying Sun, Hui Xiong

Organizations: Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou) · AI2 Robotics · Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR

Abstract

Flow-matching policies hold great promise for reinforcement learning (RL) by capturing complex, multi-modal action distributions. However, their practical application is often hindered by prohibitive inference latency and ineffective online exploration. Although recent works have employed one-step distillation for fast inference, the structure of the initial noise distribution remains an overlooked factor that presents significant untapped potential. This overlooked factor, along with the challenge of controlling policy stochasticity, constitutes two critical areas for advancing distilled flow-matching policies. To overcome these limitations, we propose GoldenStart (GSFlow), a policy distillation method with Q-guided priors and explicit entropy control. Instead of initializing generation from uninformed noise, we introduce a Q-guided prior modeled by a conditional VAE. This state-conditioned prior repositions the starting points of the one-step generation process into high-Q regions, effectively providing a "golden start" that shortcuts the policy to promising actions. Furthermore, for effective online exploration, we enable our distilled actor to output a stochastic distribution instead of a deterministic point. This is governed by entropy regularization, allowing the policy to shift from pure exploitation to principled exploration. Our integrated framework demonstrates that by designing the generative startpoint and explicitly controlling policy entropy, it is possible to achieve efficient and exploratory policies, bridging the generative models and the practical actor-critic methods. We conduct extensive experiments on offline and online continuous control benchmarks, where our method significantly outperforms prior state-of-the-art approaches. Code will be available at https://github.com/ZhHe11/GSFlow-RL.

Figures & tables

Figure 1: An illustration of denoising from an uninformed Gaussian prior (a) versus an informed, value-guided prior (b). Deeper blue indicates higher value.

Figure 2: The visualization of the multi-crescent task.

Figure 3: Overview of our algorithm. During training, we first learn a structured prior for the initial noise, which is then used to distill the teacher policy. For online exploration, actions are sampled from the student’s entropy-regularized distribution. During evaluation, the deterministic mean of the policy’s output is used. The critic update steps are omitted for clarity, detailed in Appendix B .

Figure 4: Visualization of the learned prior distribution after different training stages.

Panel of Figure 5 — Figure 5: Results on the multi-crescent task. Blue crosses denote samples from the offline dataset, while yellow stars represent the actions produced by the policies. (a): shows the offline dataset, which excludes the two globally optimal modes. (b, c): shows action distributions after the offline phase. Our method captures the higher-value modes within the dataset, while the baseline shows a less focused distribution. (d, e): shows action distributions after the online fine-tuning phase. Our method quickly discovers and converges to both highest-reward modes. In contrast, the baseline only finds one. More results on this task can be found in Appendix 8 .

OGBench
Task	Gaussian Policies			Diffusion Policies			Flow Policies
Task	BC	IQL	ReBRAC	IDQL	SRPO	CAC	FAWAC	FBRAC	IFQL	FQL	Ours
AntMaze Large Navigate	$0\pm 0$	$48\pm 9$	$\textbf{91}\pm 10$	$0\pm 0$	$0\pm 0$	$42\pm 7$	$1\pm 1$	$70\pm 20$	$24\pm 17$	$80\pm 8$	$88.4\pm 2.7$
AntMaze Giant Navigate	$0\pm 0$	$0\pm 0$	$\textbf{27}\pm 22$	$0\pm 0$	$0\pm 0$	$0\pm 0$	$0\pm 0$	$0\pm 1$	$0\pm 0$	$4\pm 5$	$10.4\pm 5.9$
HumanoidMaze Medium	$1\pm 0$	$32\pm 7$	$16\pm 9$	$1\pm 1$	$0\pm 0$	$38\pm 19$	$6\pm 2$	$25\pm 8$	$\textbf{69}\pm 19$	$19\pm 12$	$45.0\pm 19.7$
HumanoidMaze Large	$0\pm 0$	$3\pm 1$	$2\pm 1$	$0\pm 0$	$0\pm 0$	$1\pm 1$	$0\pm 0$	$0\pm 1$	$6\pm 2$	$\textbf{7}\pm 6$	$4.6\pm 4.4$

Table 1: Offline performance on OGBench and D4RL benchmarks, averaged over 5 seeds (3 for Visual Environments due to computational cost). Best results are in bold . The performance of baseline methods is reported from Park et al. (2025b) .

Task	IQL	ReBRAC	Cal-QL	RLPD	IFQL	FQL	Ours
HumanoidMaze Medium	$21\pm 13\rightarrow 16\pm 8$	$16\pm 20\rightarrow 1\pm 1$	$0\pm 0\rightarrow 0\pm 0$	$0\pm 0\rightarrow 8\pm 10$	$56\pm 35\rightarrow\textbf{82}\pm 20$	$12\pm 7\rightarrow 22\pm 12$	$45\pm 20\rightarrow 67\pm 6$
AntSoccer Arena	$2\pm 1\rightarrow 0\pm 0$	$0\pm 0\rightarrow 0\pm 0$	$0\pm 0\rightarrow 0\pm 0$	$0\pm 0\rightarrow 0\pm 0$	$26\pm 15\rightarrow 39\pm 10$	$28\pm 8\rightarrow\textbf{86}\pm 5$	$46\pm 10\rightarrow 77\pm 9$
Cube Double Play	$0\pm 1\rightarrow 0\pm 0$	$6\pm 5\rightarrow 28\pm 28$	$0\pm 0\rightarrow 0\pm 0$	$0\pm 0\rightarrow 0\pm 0$	$12\pm 9\rightarrow 40\pm 5$	$40\pm 11\rightarrow 92\pm 3$	$51\pm 6\rightarrow\textbf{99}\pm 1$
Scene Play	$14\pm 11\rightarrow 10\pm 9$	$55\pm 10\rightarrow\textbf{100}\pm 0$	$1\pm 2\rightarrow 50\pm 53$	$0\pm 0\rightarrow\textbf{100}\pm 0$	$0\pm 1\rightarrow 60\pm 39$	$82\pm 11\rightarrow\textbf{100}\pm 1$	$88\pm 9\rightarrow\textbf{100}\pm 0$
Puzzle-4x4 Play	$5\pm 2\rightarrow 1\pm 1$	$8\pm 4\rightarrow 14\pm 35$	$0\pm 0\rightarrow 0\pm 0$	$0\pm 0\rightarrow\textbf{100}\pm 1$	$23\pm 6\rightarrow 19\pm 33$	$8\pm 3\rightarrow 38\pm 52$	$17\pm 4\rightarrow\textbf{100}\pm 0$
Average	$8.4\rightarrow 5.4$	$17.0\rightarrow 28.6$	$0.2\rightarrow 10.0$	$0.0\rightarrow 41.6$	$23.4\rightarrow 48.0$	$34.0\rightarrow 67.6$	$49.4\rightarrow\textbf{88.6}$

Table 2: Offline-to-online performance comparison. Similar to Table 1 , we report the results over 5 seeds. The best online results are highlighted in bold .

Panel of Figure 6 — Figure 6: Ablation studies on the offline-to-online transition. (a): The plot analyzes the impact of the candidate number on learning efficiency. (b, c): The plots demonstrate the effectiveness of our controllable entropy, showing significant performance gains of our full method over a deterministic variant in both the Multi-Crescent environment and the Puzzle-4x4 task.

Figure 7: Average step time required on cube-double task.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Additional Studies in the Multi-Crescent Environment

Figure 9: Visualization of the OGBench tasks.

Figure 10: Visualization of the D4RL tasks.

Hyperparameter	Value
Offline Steps	1,000,000
Online Steps	1,000,000
Seed	0,2,4,8,16
Latent Dimension	8 (default)
KL Weight	0.1 (default)
Reconstruction Weight	1 (default)

Appendix

Table 3: Hyperparameters for the cube-double experiment.

Hyperparameter	Value
Offline Steps	1,000,000
Online Steps	1,000,000
Seed	0,2,4,8,16
Latent Dimension	8 (default)
KL Weight	0.1 (default)
Reconstruction Weight	1 (default)

Appendix

Table 4: Hyperparameters for the Puzzle 3x3 experiment.

Figure 11: The effect of the VAE’s latent dimension on the final return in both offline (black) and online (blue) settings.

Method	IFQL ( $N=24$ )	IFQL ( $N=28$ )	IFQL ( $N=32$ )	Ours
Reward (Avg)	7.6	8.1	8.0	9.2
Reward (Std)	1.5	1.6	1.5	0.2

Appendix

Table 5: Comparison of stability and precision between IFQL and Ours (Reward Avg/Std).

Method	IFQL	FQL	FQL-RS ( $N=32$ )	Ours
Success Rate (Avg)	9	36	37	51
Success Rate (Std)	5	6	7	6

Appendix

Table 6: Success Rate (%) on the Cube-Double task, comparing Ours with FQL and FQL-RS.

Model Setting	Ours ( $\alpha_{1}=0$ )	Ours ( $\alpha_{1}=1$ )	Ours ( $\alpha_{1}=10$ )	Ours ( $\alpha_{1}=100$ )	Baseline ( $\alpha_{1}=100$ )
Bias ( $Q-r$ )	10.0	7.0	1.1	0.4	0.3
Abs Bias ( $\|Q-r\|$ )	10.0	7.0	1.1	0.4	0.3
Return	-	-	-	9.3	7.0

Appendix

Table 7: Analysis of Q-Overestimation and Robustness. Corresponding trends and performance comparisons are also illustrated in Figure 12 .

Figure 12: Q-bias suppression and performance gain.

Setting	Target Entropy	Return	Final $\alpha_{2}$	VAE Loss
Ours (High Exp.)	High ( $mult=0.1$ )	14.0	3.6	0.090
Ours (Med Exp.)	Medium ( $mult=0.5$ )	12.8	2.3	0.093
Ours (Low Exp.)	Low ( $mult=0.75$ )	9.3	1.9	0.117
Ours w/o Q-Prior	High ( $mult=0.1$ )	5.9	2.1	-

Appendix

Table 8: Interaction analysis between entropy and Q-guided prior on Multi-Crescent Environment. Note that variations in VAE Loss are primarily driven by the KL divergence component.

Figure 13: Exploration capability on the Puzzle-4x4 task. Ours reaches states with significantly higher max_button_state , indicating a wider exploration of the state space.

Task	SAC (Off $\rightarrow$ On)	RLPD (Off $\rightarrow$ On)	Ours (Off $\rightarrow$ On)
HumanoidMaze Medium	0 $\rightarrow$ 2	0 $\rightarrow$ 84	5 $\rightarrow$ 67
AntSoccer Arena	0 $\rightarrow$ 0	0 $\rightarrow$ 0	46 $\rightarrow$ 77
Cube Double Play	0 $\rightarrow$ 0	0 $\rightarrow$ 0	51 $\rightarrow$ 99
Scene Play	0 $\rightarrow$ 100	0 $\rightarrow$ 100	88 $\rightarrow$ 100
Puzzle-4x4 Play	0 $\rightarrow$ 0	0 $\rightarrow$ 100	17 $\rightarrow$ 100
AntMaze U-Maze	0 $\rightarrow$ 100	0 $\rightarrow$ 98	100 $\rightarrow$ 100

Appendix

Table 9: Performance comparison (Success Rate %) during the Offline

\rightarrow

Online transition. All results are averaged over 3 random seeds.

Policy Distillation

Policy distillation transfers behavior from expert or larger policies into compact, deployable learners while preserving task performance.

See all 48 papers

Sep 14, 2026

Refinement-Based Flow Policy Optimization

Flow-based policies offer an expressive representation for online reinforcement learning, but conventional flow matching requires samples drawn from the distribution to be modeled. This poses a challenge when the desired action distribution is defined only implicitly by a Q-function, since directly sampling actions from the resulting distribution is generally intractable. We propose Refinement-Based Flow Policy Optimization (RFPO), a novel framework for training a flow policy in online reinforcement learning by alternating between Q-guided sample refinement and self-target flow matching. RFPO first generates actions from Gaussian noise using the current flow policy and then uses a finite-step stochastic refinement procedure to move them toward an energy-based distribution induced by the Q-function. Each refined action is then paired with its corresponding initial noise sample and used as a fixed target for flow-matching training. By repeatedly refining its own outputs and learning from the resulting targets, RFPO incorporates Q-guidance into the policy without requiring direct samples from the target distribution, while retaining the capacity to represent multiple action modes. We further provide a theoretical analysis of the distributional dynamics induced by RFPO. Across six continuous-control tasks, RFPO matches or outperforms a standard Gaussian-policy baseline on almost every task. Experiments on six synthetic two-dimensional target distributions with diverse geometries demonstrate that RFPO captures complex multimodal structure without mode collapse.

Bumgeun Park, Hyukjun Yang, Donghwan Lee

Korea Advanced Institute of Science and Technology (KAIST)

RL Control Flow Matching Policy Optimization Continuous Control Flow-Based Policy Learning

Ask PDF

Oct 9, 2026

Reinforcement Learning via Fixed-point Flow Matching

In maximum entropy and behavior-constrained reinforcement learning (RL), the optimal policy is given by a Boltzmann distribution. Generating samples from this distribution can be challenging, as it is often complex and multimodal. While flow policies trained with flow matching provide a powerful framework for representing such distributions, they traditionally rely on access to training data, which is generally unavailable in online RL. To address this limitation, we propose fixed-point flow matching (FFM), a data-free procedure for training and fine-tuning flow policies in RL. FFM leverages the target flow identity to formulate the actor update as a fixed-point iteration, utilizing a flow-matching-style regression objective where the critic's action gradients serve as regression targets. We show that the optimal policy is a fixed point of this iteration. Through extensive experiments, we demonstrate that FFM effectively fine-tunes pretrained flow policies in the offline-to-online setting and enables efficient and accurate training of flow policies from scratch in online RL. These results establish fixed-point flow matching as a principled approach for learning expressive flow-based policies without requiring training data.

Denis Blessing, Onur Celik, Gerhard Neumann

Karlsruhe Institute of Technology (KIT) Autonomous Learning Robots Lab (ALR)

Flow-Based Policy Learning Reinforcement Learning Flow Matching Maximum Entropy RL Flow-Based Generative Modeling

Ask PDF

Feb 20, 2026

Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning

Generative models have recently demonstrated remarkable success across diverse domains, motivating their adoption as expressive policies in reinforcement learning (RL). While they have shown strong performance in offline RL, particularly where the target distribution is well defined, their extension to online fine-tuning has largely been treated as a direct continuation of offline pre-training, leaving key challenges unaddressed. In this paper, we propose Flow Matching with Injected Noise for Offline-to-Online RL (FINO), a novel method that leverages flow matching-based policies to enhance sample efficiency for offline-to-online RL. FINO facilitates effective exploration by injecting noise into policy training, thereby encouraging a broader range of actions beyond those observed in the offline dataset. In addition to exploration-enhanced flow policy training, we combine an entropy-guided sampling mechanism to balance exploration and exploitation, allowing the policy to adapt its behavior throughout online fine-tuning. Experiments across diverse, challenging tasks demonstrate that FINO consistently achieves superior performance under limited online budgets.

Yongjae Shin, Jongseong Chae, Jongeui Park +1

KAIST

Offline-To-Online RL Sample-Efficient RL RL Exploration Flow Matching

Ask PDF

Abstract

Policy Distillation

Explore similar work

Refinement-Based Flow Policy Optimization

Reinforcement Learning via Fixed-point Flow Matching

Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning