Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints

Ting Gao; Stavros Orfanoudakis; Nan Lin; Winnie Daamen; Serge Hoogendoorn; Elvin Isufi

cs.LGMar 18, 2026

Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints

Authors: Ting Gao, Stavros Orfanoudakis, Nan Lin, Winnie Daamen, Serge Hoogendoorn, Elvin Isufi

Abstract

Balancing policy expressiveness with the exploration-exploitation trade-off is a core challenge in online Reinforcement Learning (RL). While Stochastic Differential Equation (SDE)-based diffusion policies can represent complex, multimodal action distributions, they suffer from two critical limitations: their stochastic reverse processes render entropy intractable (necessitating heuristic exploration), and computing policy gradients through long denoising chains is expensive and unstable. In this work, we show that ODE-based flow matching inherently resolves these issues by enabling both simulation-free policy optimization and tractable entropy computation. Building on this, we introduce Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints (FMER). Our framework exploits this insight in three ways. First, we derive an analytic entropy objective that corrects for the density distortion caused by the $\tanh$ transformation (mapping an unbounded latent space to bounded actions), thereby facilitating entropy-constrained policy optimization. Second, we theoretically establish that minimizing an advantage-weighted conditional flow matching loss acts as a simulation-free surrogate for policy mirror descent. This steers the velocity field toward high-value regions while entirely avoiding backpropagation through the ODE solver. Finally, we dynamically tune the mirror descent temperature based on the effective sample size to enforce a robust trust region during training. Empirical evaluations demonstrate that FMER achieves superior performance on the challenging sparse-reward FrankaKitchen environment, while maintaining competitive results across standard dense-reward MuJoCo benchmarks. Code is available at https://github.com/gtanaelle/FMER.git.

Figures & tables

Method	Policy Mirror Descent	Entropy Constrained	Unbiased Entropy Objective	Simulation-Free Training
DIPO [ 38 ]	✗	✗	✗	✓
DPMD [ 21 ]	✓	✗	✗	✓
QVPO [ 6 ]	✗	✓	✗	✓
SAC-Flow [ 42 ]	✗	✓	✗	✗
FLAME [ 17 ]	✗	✓	✗	✗
FMER (Ours)	✓	✓	✓	✓

Table 1: Conceptual comparison between FMER and diffusion- and flow-based baselines.

Figure 1: Overview of the FMER framework. Left (Policy Training): The policy generates a set of candidate actions, which are evaluated by the critic (Q-network) to compute advantage-based weights. These weights guide the Weighted Conditional FM loss, while an entropy regularization term ensures exploration. Right (Interaction): During data collection, the agent samples multiple candidates and executes the highest-value action to store in the replay buffer.

Figure 2: Weighted vs. greedy policy updates. Contours show the ground truth Q-value landscape for a fixed state. From 16 sampled candidate actions, the top-1 (greedy) update moves toward the highest-valued sample (red arrow), leading to a local maximum. In contrast, the weighted update (green) aggregates all candidates via Q-based weights (small arrows) towards the global optimal.

Figure 3: State-conditioned policy visualization on the 2D multi-goal task. Background colors display the ground-truth state-value landscape, with bright areas marking optimal goals. Red arrows show the selected actions for SAC [ 11 ] , DPMD [ 21 ] , and FMER, illustrating each method’s navigation strategy.

Panel of Figure 4 — Figure 4: Agents working in the default FrankaKitchen environment with a budget of 280 steps. Top row (a-e): FMER is able to finish 5 tasks. Bottom row (f-h): The best baseline model (DIPO) is able to finish 3 tasks. In this visualization, we use the best seed for both FMER and DIPO.

	FrankaKitchen (Task Completion Rate)				MuJoCo-v5
Method	N=1	N=2	N=4	N=7	Hopper	HalfCheetah	Walker	Humanoid
PPO	2.00 (4.47)	0.00 (0.00)	1.00 (2.24)	8.57 (7.82)	1198.02 (360.64)	5982.74 (810.60)	4598.21 (222.17)	811.94 (78.20)
SAC	40.00 (54.77)	32.00 (24.90)	2.50 (3.54)	12.29 (4.24)	3318.64 (132.80)	10295.33 (1170.43)	4441.51 (536.85)	5326.73 (58.93)
TD3	0.00 (0.00)	11.00 (24.60)	15.00 (17.32)	5.71 (7.82)	3662.14 (61.58)	9496.59 (901.14)	4887.65 (630.96)	318.33 (208.02)
DIPO	0.00 (0.00)	33.00 (30.33)	14.50 (21.39)	23.14 (18.36)	3408.50 (297.31)	9516.47 (1417.20)	5028.66 (217.36)	5087.46 (100.04)
DPMD	60.00 (54.77)	50.00 (0.00)	25.00 (17.68)	20.57 (8.67)	3397.57 (278.06)	10706.06 (817.59)	4901.24 (351.59)	5084.60 (27.90)

Table 2: Normalized task completion rate on FrankaKitchen environment and performance on MuJoCo-v5 environment. Values are presented as Mean (Std) across 5 random seeds. For each seed, the policy is evaluated over 10 independent episodes.

Method	$N=1$	$N=2$	$N=4$	$N=7$
DPMD, default	60.00 (54.77)	50.00 (0.00)	25.00 (17.68)	20.57 (8.67)
FMER, default	80.00 (44.72)	48.00 (4.47)	46.50 (3.79)	40.29 (21.03)
DPMD, no argmax	0.00 (0.00)	0.00 (0.00)	5.00 (11.18)	0.86 (1.28)
FMER, no argmax	100.00 (0.00)	50.00 (35.36)	26.50 (2.24)	14.29 (0.00)

Table 3: Ablation on best-of-

M

action selection on FrankaKitchen. Task completion rates (%) are presented as Mean (Std).

Figure 5: Training curves (left) and entropy evolution (right) when removing the entropy constraint after $m$ steps on Walker2d-v5.

Figure 6: Training curves comparing advantage-based weight and uniform weight on Walker2d-v5.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameters	FLAME	DIPO	DPMD	QVPO	SAC-Flow	SAC	TD3	PPO
No. of hidden layers	3	3	3	2	Multiple	2	2	2
No. of hidden nodes	256	256	256	256	Multiple	256	256	256
Activation	mish	mish	mish	mish	Multiple	ReLU	ReLU	ReLU
Batch size	256	256	256	256	512	256	256	256
Discount for reward $\gamma$	0.99	0.99	0.99	0.99	0.99	0.99	0.99	0.99
Reward scale	0.2	0.2	0.2	1	1	1	1	1

Appendix

Table 4: Hyperparameters used in the baselines.

\dagger

20 for training/sampling, 1 for test. *4 for critic update, 32 for evaluation, 64 for actor update. Note: SAC-Flow utilizes multiple sub-networks for the actor with different depths and activations.

Hyperparameters	Value
No. of hidden layers	3
No. of hidden nodes	256
Activation	mish
Batch size	256
Discount for reward $\gamma$	0.99
Target smoothing coefficient	0.005

Appendix

Table 5: Hyperparameters used in FMER (shared across all environments).

Figure 7: Policy evolution on the 2D multi-goal task for the same state $s=(0,0)$ . Background contours depict the learned critic $Q(s,a)$ over action space. For SAC, points show policy samples, and for the generative policies, red dots show the random noise samples and their sampling trajectory towards the final action.

Tasks	Description
“top burner"	Turn the oven knob that activates the top burner
“bottom burner"	Turn the oven knob that activates the bottom burner
“light switch"	Turn on the light switch
“slide cabinet"	Open the slide cabinet
“hinge cabinet"	Open the left hinge cabinet
“microwave"	Open the microwave door

Appendix

Table 6: FrankaKitchen task description.

Tasks ( $N$ )	Composition
$N=1$	“light switch"
$N=2$	“light switch", “slide cabinet"
$N=4$	“light switch", “slide cabinet", “bottom burner", “microwave"
$N=7$	“light switch", “slide cabinet", “bottom burner", “microwave", “kettle", “top burner", “hinge cabinet"

Appendix

Table 7: FrankaKitchen task compositions.

	Accomplished Tasks				Normalized Task Completion Rate (%)
Method	N=1	N=2	N=4	N=7	N=1	N=2	N=4	N=7
PPO	0.02 (0.04)	0.00 (0.00)	0.04 (0.09)	0.60 (0.55)	2.00 (4.47)	0.00 (0.00)	1.00 (2.24)	8.57 (7.82)
SAC	0.40 (0.55)	0.64 (0.50)	0.10 (0.14)	0.86 (0.30)	40.00 (54.77)	32.00 (24.90)	2.50 (3.54)	12.29 (4.24)
TD3	0.00 (0.00)	0.22 (0.49)	0.60 (0.69)	0.40 (0.55)	0.00 (0.00)	11.00 (24.60)	15.00 (17.32)	5.71 (7.82)
DIPO	0.00 (0.00)	0.66 (0.61)	0.58 (0.86)	1.62 (1.29)	0.00 (0.00)	33.00 (30.33)	14.50 (21.39)	23.14 (18.36)
DPMD	0.60 (0.55)	1.00 (0.00)	1.00 (0.71)	1.44 (0.61)	60.00 (54.77)	50.00 (0.00)	25.00 (17.68)	20.57 (8.67)

Appendix

Table 8: We report the mean number of accomplished tasks and the normalized task completion rate (%). Results reflect the maximum evaluation return achieved during the final 10% of training (steps 900K to 1M), averaged across 5 random seeds. Each evaluation point is computed over 10 independent environment episodes. Standard deviations are provided in parentheses.

Figure 8: Training curves on four MuJoCo locomotion benchmarks. We report the average episodic return over 5 random seeds, with solid lines representing the mean and shaded regions indicating the standard deviation. For visual clarity, we compare FMER against the five top-performing baselines (QVPO, DPMD, DIPO, FLAME, SAC-Flow). Curves are smoothed with a 10,000-step window.

Panel of Figure 9 — Figure 9: Training dynamics of FMER on Humanoid-v5, averaged over five seeds; shaded regions denote one standard deviation.

Algorithm	Training time	Inference time
DIPO (JAX)	294 min	2.268 ms
DPMD (JAX)	276 min	1.125 ms
QVPO (PyTorch)	19.7 h	1.284 ms
SAC-Flow (JAX)	126 min	0.483 ms
FMER (JAX)	215 min	0.515 ms

Appendix

Table 9: Total training time and inference time of 1 million environment interactions on Humanoid-v5.

Environment	Action dimension $d$	Training time	Inference time
Hopper-v5	3	167	0.471
HalfCheetah-v5	6	168	0.479
Walker2d-v5	6	171	0.480
Humanoid-v5	17	215	0.515

Appendix

Table 10: Training time (minutes) and inference time (ms) across environments with varying action dimensions (

d

), averaged over 5 random seeds.

Setting	Task completion rate (%)	Final entropy
Default FMER	40.29 (21.03)	-8.71 (1.06)
Entropy removed after $5\times 10^{5}$ steps	40.29 (19.73)	-10.43 (2.36)
Entropy removed from the start	0.00 (0.00)	NaN
Entropy removed from the start, $\tau$ clipped	0.00 (0.00)	-201.36 (11.91)

Appendix

Table 11: Entropy ablation on FrankaKitchen (

N=7

). Task completion rates are reported in percentages. NaN denotes a non-finite entropy estimate.

Setting	Task completion rate (%)
Default FMER	40.29 (21.03)
Uniform weights	33.71 (18.23)

Appendix

Table 12: Advantage-weighting ablation on FrankaKitchen (

N=7

Method	Hopper	HalfCheetah	Walker	Humanoid
clipping	2704.55 (627.12)	7701.50 (598.25)	4733.82 (696.37)	4378.76 (2021.24)
$\tanh$	2913.45 (642.67)	12332.38 (903.67)	5282.89 (437.57)	5286.12 (65.35)

Appendix

Table 13: Comparison of clipping and tanh transformations across different environments. Values represent mean returns with standard deviations in parentheses.

Method	Performance	Time (min)
Hutchinson	5282.89 (437.57)	171
Exact	5316.47 (254.97)	258

Appendix

Table 14: Comparison of Hutchinson’s estimator and exact trace computation on Walker2d-v5.

	FrankaKitchen (finished task)				MuJoCo-v5
Method	N=1	N=2	N=4	N=7	Hopper	HalfCheetah	Walker	Humanoid
$\tau=0.2$	0.20 (0.45)	0.98 (0.67)	1.00 (0.67)	3.10 (1.21)	2927.44 (531.97)	11632.94 (859.00)	5025.65 (350.93)	5399.06 (152.65)
$\tau=0.5$	0.60 (0.55)	1.42 (0.39)	1.36 (0.41)	2.84 (1.44)	2972.15 (344.36)	10888.09 (1873.16)	5504.34 (144.18)	5310.64 (87.56)
$\tau=1.0$	0.00 (0.00)	0.42 (0.58)	0.56 (0.88)	1.82 (1.10)	3222.88 (289.04)	10920.01 (2624.63)	5430.12 (293.93)	5228.06 (148.65)
Dynamic	0.80 (0.45)	0.96 (0.09)	1.86 (0.15)	2.82 (1.47)	2913.45 (642.67)	12332.38 (903.67)	5282.89 (437.57)	5286.12 (65.35)

Appendix

Table 15: Sensitivity of FMER to temperature

\tau

: fixed values versus

\mathrm{ESS}

-based dynamic tuning.

	Walker2d-v5		Humanoid-v5		Franka (N=2)		Franka (N=7)
Candidate Size	Time	Perf	Time	Perf	Time	Perf	Time	Perf
4	151	5321.36	167	5243.07	192	1.00	193	2.50
8	171	5282.89	215	5286.12	220	0.96	217	2.82
16	225	4889.52	285	5129.35	275	1.40	274	2.90
32	320	4657.87	422	5048.57	369	1.24	370	1.96

Appendix

Table 16: Performance and training time (minutes) across candidate sizes

M

and environments (5 random seeds).

$N$	Time	Perf
5	147	5493.64 (282.53)
10	171	5282.89 (437.57)
20	226	5591.64 (153.13)
50	409	5186.86 (519.63)

Appendix

Table 17: Training time (minutes) and performance on Walker2d across ODE integration steps (

N

), averaged over 5 random seeds. Performance is presented as Mean (Std).

ODE solver	ODE steps	NFE	Time	Perf
Euler	10	10	171	5282.89 (437.57)
Runge–Kutta 2	10	20	219	5519.10 (395.64)
Runge–Kutta 4	10	40	340	5486.24 (184.02)
Runge–Kutta 2	5	10	168	5234.53 (172.47)
Runge–Kutta 4	2	8	164	3988.16 (2367.62)

Appendix

Table 18: Training time (minutes) and performance on Walker2d across ODE solvers, averaged over 5 random seeds. Performance is presented as Mean (Std).

Policy Optimization

Policy optimization develops algorithms for improving decision rules in reinforcement learning, language models, and sequential decision systems.

See all 224 papers

Reinforcement Learning via Fixed-point Flow Matching
Oct 9, 2026Denis Blessing, Onur Celik, Gerhard NeumannFlow-Based Policy Learning Reinforcement Learning
Karlsruhe Institute of Technology (KIT) Autonomous Learning Robots Lab (ALR)
Entropy-Regularized Adjoint Matching for Offline Reinforcement Learning
May 7, 2026Abdelghani Ghanem, Mounir GhoghoFlow Matching Offline RL
College of Computing Mohammed VI Polytechnic University Rabat 11103, Morocco
DeFlow: Decoupling Behavior-Prior Modeling and Value Maximization for Offline Policy Extraction
Jan 15, 2026Zhancun Mu, Chi ZhangOffline RL Policy Learning
Peking University, Institute of Artificial Intelligence, China, Beijing.

Oct 9, 2026J/K move · Enter open · S save

Reinforcement Learning via Fixed-point Flow Matching

Denis Blessing, Onur Celik, Gerhard Neumann

Karlsruhe Institute of Technology (KIT) Autonomous Learning Robots Lab (ALR)

Flow-Based Policy Learning Reinforcement Learning Flow Matching Maximum Entropy RL Flow-Based Generative Modeling

Open paper AskPDF arXiv

In maximum entropy and behavior-constrained reinforcement learning (RL), the optimal policy is given by a Boltzmann distribution. Generating samples from this distribution can be challenging, as it is often complex and multimodal. While flow policies trained with flow matching provide a powerful framework for representing such distributions, they traditionally rely on access to training data, which is generally unavailable in online RL. To address this limitation, we propose fixed-point flow matching (FFM), a data-free procedure for training and fine-tuning flow policies in RL. FFM leverages the target flow identity to formulate the actor update as a fixed-point iteration, utilizing a flow-matching-style regression objective where the critic's action gradients serve as regression targets. We show that the optimal policy is a fixed point of this iteration. Through extensive experiments, we demonstrate that FFM effectively fine-tunes pretrained flow policies in the offline-to-online setting and enables efficient and accurate training of flow policies from scratch in online RL. These results establish fixed-point flow matching as a principled approach for learning expressive flow-based policies without requiring training data.

Abstract

Policy Optimization

Explore similar work

Reinforcement Learning via Fixed-point Flow Matching

Entropy-Regularized Adjoint Matching for Offline Reinforcement Learning

DeFlow: Decoupling Behavior-Prior Modeling and Value Maximization for Offline Policy Extraction