A Survey of Flow Matching in Reinforcement Learning

Nabuat Zaman Nahim; Fairoz Nower Khan; Peizhong Ju

cs.LGOct 12, 2026

A Survey of Flow Matching in Reinforcement Learning

Authors: Nabuat Zaman Nahim, Fairoz Nower Khan, Peizhong Ju

Organizations: Department of Computer Science, University of Kentucky, Lexington, Kentucky 40506, USA

Abstract

Flow Matching (FM) has recently emerged as a principled and efficient generative modeling framework for reinforcement learning (RL), enabling expressive, multimodal policy parameterizations via deterministic probability transport. Compared to diffusion-based policies that rely on stochastic denoising chains, FM uses sampling based on ordinary differential equations (ODEs), with learned velocity fields, which can substantially reduce inference latency and simplify the incorporation of RL objectives. As research in flow-based RL rapidly accelerates across offline continuous control, online fine-tuning, and foundation model alignment, the literature has become highly fragmented. In this survey, we provide a comprehensive taxonomy of flow-matching approaches in reinforcement learning. We organize the literature along two axes: the target distribution being modeled (e.g., action policies, value critics, transition dynamics) and the mechanism of RL signal integration (e.g., energy-weighted regression, flow-based policy gradients, and group relative policy optimization). Furthermore, we survey emerging frontiers such as discrete and non-Euclidean action spaces, provide a systematic comparative analysis against Gaussian and diffusion baselines, and outline critical open problems. Ultimately, this survey serves as a foundational roadmap for the next generation of generative reinforcement learning and alignment.

Figures & tables

Panel of Figure 1 — Figure 1: Flow Matching in a nutshell. Top: flow matching learns approximately straight probability transport from a simple base distribution to the target data distribution. Bottom left: training is simulation-free: one samples $(x_{0},x_{1},t)$ , constructs an interpolation point $x_{t}$ , and regresses the learned velocity field $v_{\theta}(x_{t},t)$ toward the target conditional velocity $u_{t}$ . For the common linear optimal-transport path, the target velocity is simply $x_{1}-x_{0}$ . Bottom right: after training, new samples are generated by integrating the learned ODE from base noise to the target distribution.

Panel of Figure 2 — Figure 2: How flow matching is integrated with reinforcement learning. A common flow-policy backbone defines an expressive action generator through a learned velocity field and ODE sampler. In offline RL , the reinforcement signal typically enters at training time , most often by reweighting the flow-matching objective using value or energy estimates. In online RL , the signal enters through policy optimization using environment interaction, typically via stochasticized flows, pathwise gradients through the ODE, or group-based updates such as GRPO. Across both settings, additional stabilizers such as KL regularization, Wasserstein anchoring, clipping, and support constraints are often used.

	How the RL signal enters (Axis B)
What is modeled (Axis A)	Weighted regression	Direct / surrogate gradients	Group-relative (GRPO)	Other mechanisms
Actions / policies	QIPO/EFM ( Zhang et al., 2025b ) , FlowQ ( Alles et al., 2025 ) , ARFM ( Zhang et al., 2025a ) , ORW-CFM-W2 ( Fan et al., 2025b ) , ADRPO ( Fan et al., 2025c ) , RWFM ( Pfrommer et al., 2025 )	FPO ( McAllister et al., 2025 ) , PolicyFlow ( Yang et al., 2026 ) , RL-FM ( Pfrommer et al., 2025 )	Flow-GRPO ( Liu et al., 2025a ) , TempFlow ( He et al., 2025 ) , Smart-GRPO ( Yu et al., 2026 ) , SuperFlow ( Chen et al., 2025c ) , F5R-TTS ( Sun et al., 2025 )	FQL ( Park et al., 2025b ) , SSCP ( Koirala and Fleming, 2025 ) , ReFORM ( Zhang et al., 2026 ) , MAC-Flow ( Lee et al., 2025a ) (distillation) , ReinFlow ( Zhang et al., 2025c ) (noise injection) , DiffusionNFT ( Zheng et al., 2025b ) (forward-process)
Critics / values	floq ( Agrawalla et al., 2025 ) (TD regression on flow critic)	—	—	DFC ( Chen et al., 2025a ) , FlowCritic ( Zhong et al., 2025 ) , EVOR ( Espinosa-Dice et al., 2025 ) (distributional flow critics)
Dynamics / transitions	CompFlow ( Kong et al., 2025 ) (transport regression on transitions)	—	—	—
Preferences / alignment	PFM ( Kim et al., 2024 ) , FlowRL ( Zhu et al., 2025 )	—	Fine-Grained GRPO ( Zhou et al., 2025 ) , AWM ( Xue et al., 2025 ) , GRPO-Guard ( Wang et al., 2025b )	AC-Flow ( Fan et al., 2025a ) (intermediate feedback)

Table 1: Two-axis cross summary of flow-matching methods in RL. Rows indicate what distribution the flow models (Axis A) and columns indicate how the RL signal enters the flow training (Axis B). Empty cells mark under-explored combinations that represent open research directions such as GRPO-style updates for critic or dynamics flows. This table complements the learning-setting view in Figure 3 and the offline/online/beyond narrative in Sections 4 – 6 .

\sum_{i=1}^{B}\sum_{j=1}^{M}\lambda(t)\,\bar{w}_{i}^{(j)}\big\|v_{\theta}(a_{t},s_{i},t)-u_{t}(a_{t}|a_{i}^{(j)},s_{i})\big\|_{2}^{2}.

Algorithm 3 Flow-based QIPO / EFM for Offline RL ( Zhang et al., 2025b )

\big(Q_{\phi}(s,a)-(r+\gamma Q_{\bar{\phi}}(s^{\prime},a^{\prime}))\big)^{2}.

Algorithm 4 Flow Q-Learning (FQL) for Offline RL ( Park et al., 2025b )

Algorithm	How Energy Enters	OOD Control Mechanism	Inference Cost
EFM / QIPO ( Zhang et al., 2025b )	Modified target velocity field	Implicit (KL/Energy Target)	Moderate (ODE)
FlowQ ( Alles et al., 2025 )	Guided conditional path	Implicit (KL/Energy Target)	Moderate (ODE)
FQL ( Park et al., 2025b )	Policy head optimization	Explicit KL-style penalty	Very Low (1-step)
ReFORM ( Zhang et al., 2026 )	Behavior Cloning base	Structural latent reflection	Moderate (ODE)
ARFM ( Zhang et al., 2025a )	Adaptive exponential scaling	Variance-bounded weights	Moderate (ODE)
SSCP ( Koirala and Fleming, 2025 )	Vector completion target	Implicit (Generative prior)	Very Low (1-step)

Table 2: A unifying comparison of foundational offline flow-matching RL algorithms.

a^{(i)}=\Phi_{\theta}(x_{0}^{(i)},s)

Algorithm 8 Generic GRPO-style training for flow policies

\mathcal{L}_{\mathrm{CFM}}=w(x_{1})\|v_{\theta}(t,x)-u_{t}(x\mid x_{1})\|^{2}

Algorithm 9 Online Reward-Weighted CFM with Wasserstein Regularization (ORW-CFM-W2) ( Fan et al., 2025b )

w(x_{1})=F(\tau A(x_{1}))

Algorithm 10 Adaptive Divergence Regularized Policy Optimization (ADRPO) ( Fan et al., 2025c )

Method	Regularization	Sampling	Key Idea
ORW-CFM-W2	Fixed $W_{2}$ penalty	Online	Reward-weighted flow matching
ADRPO	Adaptive $W_{2}$ penalty	Online	Advantage-dependent regularization

Table 3: Comparison of online reward-weighted flow fine-tuning methods.

\mathcal{L}=(Q_{\theta}(s,a)-y)^{2}

Algorithm 11 Flow-Matching Q-Function Training (floq) ( Agrawalla et al., 2025 )

x_{t}=(1-t)x^{-}+tx^{+}

Algorithm 13 Preference Flow Matching (PFM) ( Kim et al., 2024 )

Property	Gaussian Policy	Diffusion Policy	Flow-Matching Policy
Expressivity	Unimodal	Highly Multimodal	Highly Multimodal
Transport Geometry	N/A	Stochastic (SDE)	Deterministic (ODE)
Path Curvature	N/A	Highly Curved	Straight (Rectified)
Inference Latency	1 NFE	5–100 NFE	1–10 NFE (1 if distilled)
Online Optimization	PPO / SAC (Stable)	High Variance	PPO / GRPO (Stable)
Support Bounding	Clipping	Difficult (Drift)	Exact (e.g., ReFORM)

Table 4: Comparative summary of policy classes in continuous reinforcement learning.

Method	Class	NFE	D4RL (normalized return, %)		OGBench	Action gen.	Notes / source
			Gym-Loco (avg)	AntMaze (avg)	(50-task avg)	time ( $\downarrow$ )
Gaussian / non-generative baselines (for context)
BC	Gaussian	1	51.9	16.7	2.9	–	Wang et al. (2023)
TD3+BC	Gaussian	1	75.3	27.3	–	–	Wang et al. (2023)
IQL	Gaussian + AWR	1	77.0	63.0	23.4	–	20 min train (D4RL) ( Hansen-Estruch et al., 2023 )
ReBRAC	Gaussian	1	88.5 ⋆	76.8	31.0	–	⋆ 9-task v2 avg. from Tarasov et al. (2023) Tab. 16

Table 5: Consolidated empirical comparison of diffusion-based and flow-matching offline RL methods. NFE = number of velocity-field / denoiser evaluations per action at inference. D4RL Gym-Loco is the 9-task halfcheetah/hopper/walker2d {medium, medium-replay, medium-expert} v2 average, D4RL AntMaze is the 6-task umaze/medium/large {play, diverse} average. The OGBench column reports the 50-task state-based average across 5 locomotion and 5 manipulation environments (5 tasks each), with all baselines re-implemented in a single codebase by Park et al. (2025b) Table 2. Numbers are normalized scores (%) as reported by each paper under its own protocol; this is not a head-to-head reproduction. “–” indicates the paper does not report on that benchmark, “

\downarrow

” for time means lower is better. For methods that use rejection sampling (IDQL, IFQL), NFE is reported as

T\times N

, where

T

is the number of denoising or ODE steps per candidate action and

N

is the number of candidates scored by the Q-function.

Method	Class	NFE	Reported reward	Notes / source
MuJoCo Playground / DMC, 10-task average eval reward (60M env. steps)
Gaussian PPO	Gaussian (PPO)	1	$667.8\pm 66.0$	Tuned Brax PPO baseline ( McAllister et al., 2025 )
DPPO	Diffusion + PPO	10	$652.5\pm 83.7$	10 denoising steps, $\sigma_{t}=0.05$ ( McAllister et al., 2025 )
FPO	Flow + advantage-weighted PG	10	$\mathbf{759.3\pm 45.3}$	8 $(\tau,\epsilon)$ pairs, $\epsilon$ -MSE ( McAllister et al., 2025 )
PolicyFlow	CNF + PPO + Brownian reg.	10–12	matches/exceeds FPO, $>$ DPPO $>$ PPO	8 DMC tasks at 30M steps ( Yang et al., 2026 )
OpenAI Gym, pre-trained $\to$ fine-tuned episode reward (BC pretraining $\to$ online RL)

Table 6: Online RL with flow-matching and diffusion policies. The two blocks correspond to the two evaluation protocols used in the literature. (Top) MuJoCo Playground / DeepMind Control Suite, training from scratch with 60M environment steps where values are mean evaluation reward

\pm

standard error averaged across 10 DMC tasks (5 seeds each), reported in McAllister et al. (2025) Table 1. (Bottom) OpenAI Gym MuJoCo, offline-pretrained then online fine-tuned (

\sim

\times 10^{7}

env. samples) where values are mean episode reward

\pm

std. across 3 seeds, reported in Zhang et al. (2025c) Table 4(a). Pre-trained

\to

fine-tuned shows the gain from online RL on a flow-matching policy. PolicyFlow numbers are described qualitatively because Yang et al. (2026) report results as learning curves (Figure 3) without a numeric aggregate table and their text explicitly states PolicyFlow “achieves performance comparable to or exceeding FPO in most environments, outperforming DPPO”.

Method	Steps	GenEval $\uparrow$	OCR Acc. $\uparrow$	PickScore $\uparrow$	Focus
SD3.5-M (base)	—	0.63	0.59	21.72	—
Flow-GRPO ( Liu et al., 2025a )	5,600	0.95	0.92	23.31	Baseline
TempFlow-GRPO ( He et al., 2025 )	3,800	0.97	—	—	Temporal credit
AWM ( Xue et al., 2025 )	— †	0.95	0.95 †	23.25 †	8–24 $\times$ speedup
SuperFlow ( Chen et al., 2025c )	1,846 ‡	0.80 ‡	0.84 ‡	0.87 ∗	Efficiency
Smart-GRPO ( Yu et al., 2026 )	360 §	—	—	— ∗∗	Noise optimization

Table 7: Reported results for the GRPO family on text-to-image alignment (SD3.5-M base). All numbers are taken from the respective original papers and are not reproduced under identical evaluation conditions. Empty cells indicate metrics not reported by that method on this benchmark. † AWM reports GPU hours rather than steps and reaches the listed scores with extended training (79 and 205 GPU hours for OCR and PickScore respectively). ‡ SuperFlow and GRPO-Guard evaluate at step counts matched to Flow-GRPO to demonstrate efficiency or stability gains, not converged final performance. ∗ SuperFlow reports PickScore on a different normalization scale and is not directly comparable to the raw PickScore values in other rows. ∗∗ Smart-GRPO reports ImageReward (0.86) and Aesthetic Score (6.24) instead of the three benchmarks above, evaluated on a 1,000-prompt GenEval-script set. § Smart-GRPO reports in epochs rather than steps.

Explore similar work

Reinforcement Learning via Fixed-point Flow Matching
Oct 9, 2026Denis Blessing, Onur Celik, Gerhard NeumannReinforcement Learning Flow Matching
Karlsruhe Institute of Technology (KIT) Autonomous Learning Robots Lab (ALR)
Discrete Flow Matching for Offline-to-Online Reinforcement Learning
May 12, 2026Fairoz Nower Khan, Nabuat Zaman Nahim, Peizhong JuFlow Matching Reinforcement Learning
Department of Computer Science, University of Kentucky
Aligning Flow Map Policies with Optimal Q-Guidance
May 12, 2026Christos Ziakas, Alessandra Russo, Avishek Joey BoseFlow Matching Policy Optimization
Imperial College London · Mila

Oct 9, 2026J/K move · Enter open · S save

Reinforcement Learning via Fixed-point Flow Matching

Denis Blessing, Onur Celik, Gerhard Neumann

Karlsruhe Institute of Technology (KIT) Autonomous Learning Robots Lab (ALR)

Reinforcement Learning Flow Matching Maximum Entropy RL Flow-Based Generative Modeling

Open paper AskPDF arXiv

In maximum entropy and behavior-constrained reinforcement learning (RL), the optimal policy is given by a Boltzmann distribution. Generating samples from this distribution can be challenging, as it is often complex and multimodal. While flow policies trained with flow matching provide a powerful framework for representing such distributions, they traditionally rely on access to training data, which is generally unavailable in online RL. To address this limitation, we propose fixed-point flow matching (FFM), a data-free procedure for training and fine-tuning flow policies in RL. FFM leverages the target flow identity to formulate the actor update as a fixed-point iteration, utilizing a flow-matching-style regression objective where the critic's action gradients serve as regression targets. We show that the optimal policy is a fixed point of this iteration. Through extensive experiments, we demonstrate that FFM effectively fine-tunes pretrained flow policies in the offline-to-online setting and enables efficient and accurate training of flow policies from scratch in online RL. These results establish fixed-point flow matching as a principled approach for learning expressive flow-based policies without requiring training data.

Abstract

Explore similar work

Reinforcement Learning via Fixed-point Flow Matching

Discrete Flow Matching for Offline-to-Online Reinforcement Learning

Aligning Flow Map Policies with Optimal Q-Guidance