Reinforcement Learning via Fixed-point Flow Matching

Denis Blessing; Onur Celik; Gerhard Neumann

cs.LGOct 9, 2026

Reinforcement Learning via Fixed-point Flow Matching

Authors: Denis Blessing, Onur Celik, Gerhard Neumann

Organizations: Karlsruhe Institute of Technology (KIT) Autonomous Learning Robots Lab (ALR)

Abstract

In maximum entropy and behavior-constrained reinforcement learning (RL), the optimal policy is given by a Boltzmann distribution. Generating samples from this distribution can be challenging, as it is often complex and multimodal. While flow policies trained with flow matching provide a powerful framework for representing such distributions, they traditionally rely on access to training data, which is generally unavailable in online RL. To address this limitation, we propose fixed-point flow matching (FFM), a data-free procedure for training and fine-tuning flow policies in RL. FFM leverages the target flow identity to formulate the actor update as a fixed-point iteration, utilizing a flow-matching-style regression objective where the critic's action gradients serve as regression targets. We show that the optimal policy is a fixed point of this iteration. Through extensive experiments, we demonstrate that FFM effectively fine-tunes pretrained flow policies in the offline-to-online setting and enables efficient and accurate training of flow policies from scratch in online RL. These results establish fixed-point flow matching as a principled approach for learning expressive flow-based policies without requiring training data.

Figures & tables

Panel of Figure 1 — Figure 1: The FFM policy-improvement step. FFM repeats: (1) Clean actions $X_{1}$ are generated from the current policy by integrating Equation 5 (forward only, no gradients); the grey curves are sampling trajectories. (2) Critic action gradients are required for the FFM regression target of Equation 12 . (3) Fresh noise $\varepsilon$ and the clean action $X_{1}$ are used to sample a noised action $X_{t}$ via Equation 3 . (4) A flow-matching-style regression pulls $v_{\theta}$ toward the FFM target. The optimal policy $\pi^{\star}\propto e^{Q/\lambda}$ is a fixed point of this loop.

Panel of Figure 2 — Figure 2: Per-domain aggregated offline-to-online success rate (mean over the five tasks and all seeds). Offline ( $\leq\!1$ M) and online ( $>\!1$ M) phases are shaded; FFM (ours) is drawn in bold with a $\pm 1$ standard-deviation band. Per-task curves and full tables are in Section E.3 .

Figure 3: Maximum Entropy Online RL. We have run FFM on all 20 MJX Playground tasks and on the four MJX humanoid tasks and present the aggregated results over all Playground tasks (left two panels) and the aggregated results over the humanoid tasks (right two panels). We have evaluated all methods on

10

seeds for every single environment (see Figure 4 and Figure 5 for results) and report the IQM with

95\%

stratified bootstrapped confidence intervals following Agarwal et al. (2021) . FFM performs on par with the baselines on the Playground tasks (left two panels) but converges to the highest return much faster than TRUDI on the high-dimensional humanoid tasks. This result indicates that FFM’s benefits are most pronounced on high-dimensional tasks. Furthermore, we can boost FFM’s performance by leveraging its deterministic mapping properties and incorporating correlated exploration (FFM + OU), which leads to a significantly faster convergence to a similar return value as the best-performing baseline DIME (left two panels) (see Figure 15 ).

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Maximum Entropy Online RL: MJX Playground tasks. We plot the IQM per-task learning curves with

95\%

stratified bootstrapped confidence intervals ( Agarwal et al., 2021 ) over

10

seeds. On most of the tasks, FFM performs similarly to or on par with baselines. However, we observe that there are some tasks such as AcrobotSwingupSparse, HopperHop, HopperStand , where FFM (and most of the baselines) perform badly, indicating that a more sophisticated exploration strategy is required. On a subset of the tasks, we have therefore run FFM with correlated exploration (FFM + OU), where we show the chosen

\tau_{\mathrm{ou}}

hyperparameter on the respective plot. Please note that correlated exploration is only possible due to FFM’s deterministic mapping ( Section 5 ) Correlated exploration can help improve performance, such as in HopperHop , or even more significantly in the HopperStand task. The aggregated results when using the correlated exploration curves whenever applicable are given in Figure 3 (left two panels).

Figure 5: Maximum Entropy Online RL: MJX Humanoid tasks. We plot the IQM per-task learning curves with

95\%

stratified bootstrapped confidence intervals ( Agarwal et al., 2021 ) over

10

seeds. FFM converges faster to a similar performance level as TRUDI and performs significantly better than TRUDI on the T1JoystickRoughTerrain task. The high-dimensional humanoid tasks showcase FFM’s benefits more clearly.

Actor Network
Parameter	FFM (ours)	AMDP	REPPO	DIME	SPO	PPO	FPO	DPPO
Hidden dim.	512	512	512	512	256	256	32	256
Hidden layers	3	3	3	3	3	3	5	3
Activation	GeLU	GeLU	GeLU	GeLU	ELU	ELU	SiLU	Mish
Flow/diff. steps	16	16	N/A	16	N/A	N/A	10	8
Critic Network

Appendix

Table 1: Hyperparameters of FFM and of the online baselines.

Figure 6: Ablation. Geometric versus linear interpolant schedule. As discussed in Section D.4 , FFM is not restricted to a specific interpolation schedule type such as the linear schedule. Motivated by diffusion-based methods, we have compared the performance of a linear schedule to the geometric schedule as described in Section D.4 and report the aggregated results over all 20 MJX Playground tasks (left two panels) and the four humanoid tasks (right two panels), where every single task was run on

10

seeds, and we report the IQM and the stratified bootstrapped

95\%

confidence intervals ( Agarwal et al., 2021 ) . While the results indicate that there is barely a performance difference in the Playground tasks, a slight improvement on the humanoid tasks can be seen, which is why we have chosen the geometric schedule as default for the online RL tasks. See Figure 7 and Figure 8 for the per-task results.

Figure 7: Ablation. Geometric versus linear interpolant schedule (MJX Playground). We plot the IQM per-task learning curves with

95\%

stratified bootstrapped confidence intervals ( Agarwal et al., 2021 ) over

10

seeds on the 20 MJX Playground tasks. The performance between the schedule types is the same for most tasks and differs only for some tasks such as FingerSpin or PendulumSwingup , which, however, lie within the confidence intervals. Therefore, there is no statistically clear result on which interpolant should be used for the Playground tasks.

Figure 8: Ablation. Geometric versus linear interpolant schedule (MJX Humanoid). We plot the IQM per-task learning curves with

95\%

stratified bootstrapped confidence intervals ( Agarwal et al., 2021 ) over

10

seeds on the four MJX humanoid tasks. While for the G1JoystickFlatTerrain and for the G1JoystickRoughTerrain tasks there is no difference in performance, it seems that the geometric schedule performs slightly better on the T1 tasks, which is why we have chosen the geometric schedule as default for the online RL experiments.

Figure 9: Ablation. Number of Integration Steps (left two panels) and Action Squashing mapping (right two panels). Number of integration steps (left two panels): We analyze the effect of varying integration steps and report the aggregated results over all four humanoid tasks (left two panels). Every single task was run on

10

seeds, and we report the IQM and the

95\%

stratified bootstrapped confidence intervals ( Agarwal et al., 2021 ) . The results indicate that with an increasing number of integration steps, the performance increases until it plateaus from

K=16

onwards, which is why we have chosen

K=16

as default in our experiments. Please see Figure 10 for the per-task learning curves. Action Squashing Mapping (right two panels): We analyze the effect of utilizing the

\mathrm{tanh}

and the

\mathrm{erf}

as squashing functions for mapping the generated actions to a defined action bound. We have run experiments on the four MJX humanoid tasks and report the aggregated IQM with

95\%

stratified bootstrapped confidence intervals (

10

seeds per task) ( Agarwal et al., 2021 ) in the two right panels. The results show that FFM is not sensitive to the choice of the squashing function. Please see Figure 11 for the per-task learning curves.

Figure 10: Ablation. Number of Integration Steps. We plot the IQM per-task learning curves with

95\%

stratified bootstrapped confidence intervals ( Agarwal et al., 2021 ) over

10

seeds on the four MJX humanoid tasks for varying numbers of integration steps

K

. With increasing number of integration steps, the performance improves. However, the performance plateaus at

K=16

steps.

Figure 11: Ablation. Action squashing mapping. We plot the IQM per-task learning curves with

95\%

stratified bootstrapped confidence intervals ( Agarwal et al., 2021 ) over

10

seeds on the four MJX humanoid tasks for the

\mathrm{tanh}

and

\mathrm{erf}

squashing functions. The results indicate that FFM is insensitive to the choice of the squashing function.

Figure 12: Ablation. Deterministic evaluation. We compare the effect of evaluating FFM using

X_{0}=0

(deterministic) and

X_{0}\sim\mathcal{N}(0,I)

(stochastic) on the 20 MJX Playground (left two panels) and four humanoid tasks (right two panels), where every task was run on

10

seeds, and we plot the aggregated IQM with stratified bootstrapped

95\%

confidence intervals ( Agarwal et al., 2021 ) . While there is almost no performance gap between both variants for the Playground tasks, a significant performance boost follows from the deterministic evaluation on the humanoid tasks. This indicates that deterministic evaluation matters most for high-dimensional tasks. Please see Figure 13 and Figure 14 for the per-task performances of each task suite.

Figure 13: Ablation. Deterministic evaluation (MJX Playground). We plot the IQM per-task learning curves with

95\%

stratified bootstrapped confidence intervals ( Agarwal et al., 2021 ) over

10

seeds on each of the 20 MJX Playground tasks for

X_{0}=0

(deterministic) and

X_{0}\sim\mathcal{N}(0,I)

(stochastic) during evaluation. While most of the results indicate that there are not many differences, for FingerSpin , or WalkerRun , deterministic evaluation seems to improve performance.

Figure 14: Ablation. Deterministic evaluation (MJX Humanoids). We plot the IQM per-task learning curves with

95\%

stratified bootstrapped confidence intervals ( Agarwal et al., 2021 ) over

10

seeds on each of the four MJX humanoid tasks for

X_{0}=0

(deterministic) and

X_{0}\sim\mathcal{N}(0,I)

(stochastic) during evaluation. For all four tasks, deterministic evaluation shows significantly improved performance, indicating that stochasticity can harm performance, in particular for high-dimensional action spaces. We use deterministic evaluation throughout all of our experiments.

Figure 15: Ablation. Correlated Exploration. On a subset of the MJX Playground tasks, we analyze the effect of correlated exploration by varying

\tau_{\mathrm{ou}}\in\{1,2,4,8,16,32\}

, where we have run every curve on

10

seeds and report the IQM with

95\%

stratified bootstrapped confidence intervals ( Agarwal et al., 2021 ) . Correlated exploration can significantly boost FFM’s performance for the right choice of

\tau_{\mathrm{ou}}

. For example, the HopperStand task is solved, while FFM without correlated exploration struggles on this task.

Figure 16: The ten evaluation environments (five tasks each). Top: locomotion; bottom: manipulation.

Environment	Type	State	Act	Ep. len	Dataset	Width
AntMaze-Large	locomotion	29	8	1000	1M	512
AntMaze-Giant	locomotion	29	8	1000	10M	1024
HumanoidMaze-Medium	locomotion	69	21	2000	1M	512
HumanoidMaze-Large	locomotion	69	21	2000	1M	512
Cube-Double	manipulation	37	5	500	1M	512
Cube-Triple	manipulation	46	5	1000	10M	1024

Appendix

Table 2: Environment specifications. Network width and layer norm follow the dataset-size convention (1M: width

512

; 10M/100M: width

1024

with actor layer norm), applied identically to the BC-pretraining and fine-tuning networks.

Optimizer	Adam, learning rate $3\times 10^{-4}$
Batch size	$256$
MLP	4 hidden layers, width $512$ (1M) / $1024$ (10M–100M)
Layer norm	critic: yes; actor: $1024$ -width domains only
Discount $\gamma$	$0.995$ (HumanoidMaze: $0.999$ )
Pessimism $\varrho$	$0.5$ (HumanoidMaze: $0.0$ )
Target EMA rate	$0.005$

Appendix

Table 3: Shared hyperparameters (all methods).

	Regime A (locomotion)	Regime B (manipulation)
Critic scale $1/\lambda$	$30$	$1$
Fisher normalizer	fixed ( $\bar{\mathcal{I}}=1$ )	adaptive (EMA)
Behavior anchor	trust region on	trust region off (product tilt only)
Action chunking	no	yes ( $h=5$ )

Appendix

Table 4: FFM two-regime recipe: the only per-domain choice.

Environment	FFM	TRQAM	QAM	QAM-E	IFQL	FQL	DSRL	CGQL-L
AntMaze-Large	0.89	0.89	0.62	0.86	0.29	0.38	0.53	0.48
AntMaze-Giant	0.26	0.41	0.29	0.06	0.12	0.02	0.01	0.07
HumanoidMaze-Medium	0.99	0.84	0.64	0.59	0.93	0.74	0.53	0.57
HumanoidMaze-Large	0.84	0.36	0.04	0.04	0.30	0.02	0.01	0.06
Cube-Double	0.47	0.81	0.71	0.71	0.09	0.44	0.72	0.55
Cube-Triple	0.25	0.50	0.19	0.11	0.24	0.07	0.34	0.00

Appendix

Table 5: Offline-phase success rate at the end of offline training (1M steps), mean over 5 tasks and seeds per environment. Best per row in bold.

Environment	FFM	TRQAM	QAM	QAM-E	IFQL	FQL	DSRL	CGQL-L
AntMaze-Large	1.00	0.98	0.99	0.96	0.52	1.00	0.95	0.97
AntMaze-Giant	0.98	0.97	0.74	0.97	0.44	0.84	0.90	0.61
HumanoidMaze-Medium	1.00	0.97	0.82	0.78	0.75	0.96	0.82	0.95
HumanoidMaze-Large	0.99	0.76	0.26	0.27	0.12	0.37	0.28	0.74
Cube-Double	0.99	1.00	1.00	1.00	0.72	1.00	0.99	0.99
Cube-Triple	0.99	1.00	0.90	0.98	0.46	0.87	0.90	0.20

Appendix

Table 6: Offline-to-online success rate at the end of online fine-tuning (1.5M total steps), mean over 5 tasks and seeds per environment. Best per row in bold.

Panel of Figure 18 — Figure 18: Critic scale and integration steps (offline success, mean $\pm$ std over three seeds, one manipulation and one locomotion task). Left two panels: fixed, adaptive, and adaptive-shape Fisher normalizer against $1/\lambda$ . Right two panels: number of Euler steps.

Panel of Figure 19 — Figure 19: Behavior anchor. Product tilt $p_{\mathrm{ref}}\,e^{Q/\lambda}$ vs. pure tilt $e^{Q/\lambda}$ , with the trust region on or off, offline and online (mean $\pm$ std over three seeds; the online runs use $\varepsilon_{\mathrm{TR}}=8$ ).

Panel of Figure 20 — Figure 20: Trust-region budget sweep. Success against $\varepsilon_{\mathrm{TR}}$ on four tasks (mean $\pm$ std over three seeds), offline (left two panels) and online (right two panels), for the pure tilt (first and third panel) and the product tilt (second and fourth panel). With the offline trust region off, the product tilt scores $0.41$ , $0.94$ , $1.00$ , and $0.81$ on cube-double , scene , puzzle-3x3 , and antmaze-large .

Explore similar work

Feb 20, 2026

Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning

Generative models have recently demonstrated remarkable success across diverse domains, motivating their adoption as expressive policies in reinforcement learning (RL). While they have shown strong performance in offline RL, particularly where the target distribution is well defined, their extension to online fine-tuning has largely been treated as a direct continuation of offline pre-training, leaving key challenges unaddressed. In this paper, we propose Flow Matching with Injected Noise for Offline-to-Online RL (FINO), a novel method that leverages flow matching-based policies to enhance sample efficiency for offline-to-online RL. FINO facilitates effective exploration by injecting noise into policy training, thereby encouraging a broader range of actions beyond those observed in the offline dataset. In addition to exploration-enhanced flow policy training, we combine an entropy-guided sampling mechanism to balance exploration and exploitation, allowing the policy to adapt its behavior throughout online fine-tuning. Experiments across diverse, challenging tasks demonstrate that FINO consistently achieves superior performance under limited online budgets.

Yongjae Shin, Jongseong Chae, Jongeui Park +1

KAIST

Offline-To-Online RL Sample-Efficient RL RL Exploration Flow Matching

Ask PDF

Sep 14, 2026

Refinement-Based Flow Policy Optimization

Flow-based policies offer an expressive representation for online reinforcement learning, but conventional flow matching requires samples drawn from the distribution to be modeled. This poses a challenge when the desired action distribution is defined only implicitly by a Q-function, since directly sampling actions from the resulting distribution is generally intractable. We propose Refinement-Based Flow Policy Optimization (RFPO), a novel framework for training a flow policy in online reinforcement learning by alternating between Q-guided sample refinement and self-target flow matching. RFPO first generates actions from Gaussian noise using the current flow policy and then uses a finite-step stochastic refinement procedure to move them toward an energy-based distribution induced by the Q-function. Each refined action is then paired with its corresponding initial noise sample and used as a fixed target for flow-matching training. By repeatedly refining its own outputs and learning from the resulting targets, RFPO incorporates Q-guidance into the policy without requiring direct samples from the target distribution, while retaining the capacity to represent multiple action modes. We further provide a theoretical analysis of the distributional dynamics induced by RFPO. Across six continuous-control tasks, RFPO matches or outperforms a standard Gaussian-policy baseline on almost every task. Experiments on six synthetic two-dimensional target distributions with diverse geometries demonstrate that RFPO captures complex multimodal structure without mode collapse.

Bumgeun Park, Hyukjun Yang, Donghwan Lee

Korea Advanced Institute of Science and Technology (KAIST)

RL Control Flow Matching Policy Optimization Continuous Control

Ask PDF

Oct 12, 2026

A Survey of Flow Matching in Reinforcement Learning

Flow Matching (FM) has recently emerged as a principled and efficient generative modeling framework for reinforcement learning (RL), enabling expressive, multimodal policy parameterizations via deterministic probability transport. Compared to diffusion-based policies that rely on stochastic denoising chains, FM uses sampling based on ordinary differential equations (ODEs), with learned velocity fields, which can substantially reduce inference latency and simplify the incorporation of RL objectives. As research in flow-based RL rapidly accelerates across offline continuous control, online fine-tuning, and foundation model alignment, the literature has become highly fragmented. In this survey, we provide a comprehensive taxonomy of flow-matching approaches in reinforcement learning. We organize the literature along two axes: the target distribution being modeled (e.g., action policies, value critics, transition dynamics) and the mechanism of RL signal integration (e.g., energy-weighted regression, flow-based policy gradients, and group relative policy optimization). Furthermore, we survey emerging frontiers such as discrete and non-Euclidean action spaces, provide a systematic comparative analysis against Gaussian and diffusion baselines, and outline critical open problems. Ultimately, this survey serves as a foundational roadmap for the next generation of generative reinforcement learning and alignment.

Nabuat Zaman Nahim, Fairoz Nower Khan, Peizhong Ju

Department of Computer Science, University of Kentucky, Lexington, Kentucky 40506, USA

Reinforcement Learning Flow-Based Generative Modeling RL for Generative Models

Ask PDF