Balancing policy expressiveness with the exploration-exploitation trade-off is a core challenge in online Reinforcement Learning (RL). While Stochastic Differential Equation (SDE)-based diffusion policies can represent complex, multimodal action distributions, they suffer from two critical limitations: their stochastic reverse processes render entropy intractable (necessitating heuristic exploration), and computing policy gradients through long denoising chains is expensive and unstable. In this work, we show that ODE-based flow matching inherently resolves these issues by enabling both simulation-free policy optimization and tractable entropy computation. Building on this, we introduce Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints (FMER). Our framework exploits this insight in three ways. First, we derive an analytic entropy objective that corrects for the density distortion caused by the tanh transformation (mapping an unbounded latent space to bounded actions), thereby facilitating entropy-constrained policy optimization. Second, we theoretically establish that minimizing an advantage-weighted conditional flow matching loss acts as a simulation-free surrogate for policy mirror descent. This steers the velocity field toward high-value regions while entirely avoiding backpropagation through the ODE solver. Finally, we dynamically tune the mirror descent temperature based on the effective sample size to enforce a robust trust region during training. Empirical evaluations demonstrate that FMER achieves superior performance on the challenging sparse-reward FrankaKitchen environment, while maintaining competitive results across standard dense-reward MuJoCo benchmarks. Code is available at https://github.com/gtanaelle/FMER.git.
Figures & tables
Method
Policy Mirror Descent
Entropy Constrained
Unbiased Entropy Objective
Simulation-Free Training
DIPO [ 38 ]
✗
✗
✗
✓
DPMD [ 21 ]
✓
✗
✗
✓
QVPO [ 6 ]
✗
✓
✗
✓
SAC-Flow [ 42 ]
✗
✓
✗
✗
FLAME [ 17 ]
✗
✓
✗
✗
FMER (Ours)
✓
✓
✓
✓
Table 1: Conceptual comparison between FMER and diffusion- and flow-based baselines.
Figure 1: Overview of the FMER framework. Left (Policy Training): The policy generates a set of candidate actions, which are evaluated by the critic (Q-network) to compute advantage-based weights. These weights guide the Weighted Conditional FM loss, while an entropy regularization term ensures exploration. Right (Interaction): During data collection, the agent samples multiple candidates and executes the highest-value action to store in the replay buffer.
Figure 2: Weighted vs. greedy policy updates. Contours show the ground truth Q-value landscape for a fixed state. From 16 sampled candidate actions, the top-1 (greedy) update moves toward the highest-valued sample (red arrow), leading to a local maximum. In contrast, the weighted update (green) aggregates all candidates via Q-based weights (small arrows) towards the global optimal.
Figure 3: State-conditioned policy visualization on the 2D multi-goal task. Background colors display the ground-truth state-value landscape, with bright areas marking optimal goals. Red arrows show the selected actions for SAC [ 11 ] , DPMD [ 21 ] , and FMER, illustrating each method’s navigation strategy.
Figure 4: Agents working in the default FrankaKitchen environment with a budget of 280 steps. Top row (a-e): FMER is able to finish 5 tasks. Bottom row (f-h): The best baseline model (DIPO) is able to finish 3 tasks. In this visualization, we use the best seed for both FMER and DIPO.
FrankaKitchen (Task Completion Rate)
MuJoCo-v5
Method
N=1
N=2
N=4
N=7
Hopper
HalfCheetah
Walker
Humanoid
PPO
2.00 (4.47)
0.00 (0.00)
1.00 (2.24)
8.57 (7.82)
1198.02 (360.64)
5982.74 (810.60)
4598.21 (222.17)
811.94 (78.20)
SAC
40.00 (54.77)
32.00 (24.90)
2.50 (3.54)
12.29 (4.24)
3318.64 (132.80)
10295.33 (1170.43)
4441.51 (536.85)
5326.73 (58.93)
TD3
0.00 (0.00)
11.00 (24.60)
15.00 (17.32)
5.71 (7.82)
3662.14 (61.58)
9496.59 (901.14)
4887.65 (630.96)
318.33 (208.02)
DIPO
0.00 (0.00)
33.00 (30.33)
14.50 (21.39)
23.14 (18.36)
3408.50 (297.31)
9516.47 (1417.20)
5028.66 (217.36)
5087.46 (100.04)
DPMD
60.00 (54.77)
50.00 (0.00)
25.00 (17.68)
20.57 (8.67)
3397.57 (278.06)
10706.06 (817.59)
4901.24 (351.59)
5084.60 (27.90)
Table 2: Normalized task completion rate on FrankaKitchen environment and performance on MuJoCo-v5 environment. Values are presented as Mean (Std) across 5 random seeds. For each seed, the policy is evaluated over 10 independent episodes.
Method
N=1
N=2
N=4
N=7
DPMD, default
60.00 (54.77)
50.00 (0.00)
25.00 (17.68)
20.57 (8.67)
FMER, default
80.00 (44.72)
48.00 (4.47)
46.50 (3.79)
40.29 (21.03)
DPMD, no argmax
0.00 (0.00)
0.00 (0.00)
5.00 (11.18)
0.86 (1.28)
FMER, no argmax
100.00 (0.00)
50.00 (35.36)
26.50 (2.24)
14.29 (0.00)
Table 3: Ablation on best-of- M action selection on FrankaKitchen. Task completion rates (%) are presented as Mean (Std).
Figure 5: Training curves (left) and entropy evolution (right) when removing the entropy constraint after m steps on Walker2d-v5.
Figure 6: Training curves comparing advantage-based weight and uniform weight on Walker2d-v5.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameters
FLAME
DIPO
DPMD
QVPO
SAC-Flow
SAC
TD3
PPO
No. of hidden layers
3
3
3
2
Multiple
2
2
2
No. of hidden nodes
256
256
256
256
Multiple
256
256
256
Activation
mish
mish
mish
mish
Multiple
ReLU
ReLU
ReLU
Batch size
256
256
256
256
512
256
256
256
Discount for reward γ
0.99
0.99
0.99
0.99
0.99
0.99
0.99
0.99
Reward scale
0.2
0.2
0.2
1
1
1
1
1
Appendix
Table 4: Hyperparameters used in the baselines. † 20 for training/sampling, 1 for test. *4 for critic update, 32 for evaluation, 64 for actor update. Note: SAC-Flow utilizes multiple sub-networks for the actor with different depths and activations.
Hyperparameters
Value
No. of hidden layers
3
No. of hidden nodes
256
Activation
mish
Batch size
256
Discount for reward γ
0.99
Target smoothing coefficient
0.005
Appendix
Table 5: Hyperparameters used in FMER (shared across all environments).
Figure 7: Policy evolution on the 2D multi-goal task for the same state s=(0,0) . Background contours depict the learned critic Q(s,a) over action space. For SAC, points show policy samples, and for the generative policies, red dots show the random noise samples and their sampling trajectory towards the final action.
Tasks
Description
“top burner"
Turn the oven knob that activates the top burner
“bottom burner"
Turn the oven knob that activates the bottom burner
Table 8: We report the mean number of accomplished tasks and the normalized task completion rate (%). Results reflect the maximum evaluation return achieved during the final 10% of training (steps 900K to 1M), averaged across 5 random seeds. Each evaluation point is computed over 10 independent environment episodes. Standard deviations are provided in parentheses.
Figure 8: Training curves on four MuJoCo locomotion benchmarks. We report the average episodic return over 5 random seeds, with solid lines representing the mean and shaded regions indicating the standard deviation. For visual clarity, we compare FMER against the five top-performing baselines (QVPO, DPMD, DIPO, FLAME, SAC-Flow). Curves are smoothed with a 10,000-step window.
Figure 9: Training dynamics of FMER on Humanoid-v5, averaged over five seeds; shaded regions denote one standard deviation.
Algorithm
Training time
Inference time
DIPO (JAX)
294 min
2.268 ms
DPMD (JAX)
276 min
1.125 ms
QVPO (PyTorch)
19.7 h
1.284 ms
SAC-Flow (JAX)
126 min
0.483 ms
FMER (JAX)
215 min
0.515 ms
Appendix
Table 9: Total training time and inference time of 1 million environment interactions on Humanoid-v5.
Environment
Action dimension d
Training time
Inference time
Hopper-v5
3
167
0.471
HalfCheetah-v5
6
168
0.479
Walker2d-v5
6
171
0.480
Humanoid-v5
17
215
0.515
Appendix
Table 10: Training time (minutes) and inference time (ms) across environments with varying action dimensions ( d ), averaged over 5 random seeds.
Setting
Task completion rate (%)
Final entropy
Default FMER
40.29 (21.03)
-8.71 (1.06)
Entropy removed after 5×105 steps
40.29 (19.73)
-10.43 (2.36)
Entropy removed from the start
0.00 (0.00)
NaN
Entropy removed from the start, τ clipped
0.00 (0.00)
-201.36 (11.91)
Appendix
Table 11: Entropy ablation on FrankaKitchen ( N=7 ). Task completion rates are reported in percentages. NaN denotes a non-finite entropy estimate.
Setting
Task completion rate (%)
Default FMER
40.29 (21.03)
Uniform weights
33.71 (18.23)
Appendix
Table 12: Advantage-weighting ablation on FrankaKitchen ( N=7 ).
Method
Hopper
HalfCheetah
Walker
Humanoid
clipping
2704.55 (627.12)
7701.50 (598.25)
4733.82 (696.37)
4378.76 (2021.24)
tanh
2913.45 (642.67)
12332.38 (903.67)
5282.89 (437.57)
5286.12 (65.35)
Appendix
Table 13: Comparison of clipping and tanh transformations across different environments. Values represent mean returns with standard deviations in parentheses.
Method
Performance
Time (min)
Hutchinson
5282.89 (437.57)
171
Exact
5316.47 (254.97)
258
Appendix
Table 14: Comparison of Hutchinson’s estimator and exact trace computation on Walker2d-v5.
FrankaKitchen (finished task)
MuJoCo-v5
Method
N=1
N=2
N=4
N=7
Hopper
HalfCheetah
Walker
Humanoid
τ=0.2
0.20 (0.45)
0.98 (0.67)
1.00 (0.67)
3.10 (1.21)
2927.44 (531.97)
11632.94 (859.00)
5025.65 (350.93)
5399.06 (152.65)
τ=0.5
0.60 (0.55)
1.42 (0.39)
1.36 (0.41)
2.84 (1.44)
2972.15 (344.36)
10888.09 (1873.16)
5504.34 (144.18)
5310.64 (87.56)
τ=1.0
0.00 (0.00)
0.42 (0.58)
0.56 (0.88)
1.82 (1.10)
3222.88 (289.04)
10920.01 (2624.63)
5430.12 (293.93)
5228.06 (148.65)
Dynamic
0.80 (0.45)
0.96 (0.09)
1.86 (0.15)
2.82 (1.47)
2913.45 (642.67)
12332.38 (903.67)
5282.89 (437.57)
5286.12 (65.35)
Appendix
Table 15: Sensitivity of FMER to temperature τ : fixed values versus ESS -based dynamic tuning.
Walker2d-v5
Humanoid-v5
Franka (N=2)
Franka (N=7)
Candidate Size
Time
Perf
Time
Perf
Time
Perf
Time
Perf
4
151
5321.36
167
5243.07
192
1.00
193
2.50
8
171
5282.89
215
5286.12
220
0.96
217
2.82
16
225
4889.52
285
5129.35
275
1.40
274
2.90
32
320
4657.87
422
5048.57
369
1.24
370
1.96
Appendix
Table 16: Performance and training time (minutes) across candidate sizes M and environments (5 random seeds).
N
Time
Perf
5
147
5493.64 (282.53)
10
171
5282.89 (437.57)
20
226
5591.64 (153.13)
50
409
5186.86 (519.63)
Appendix
Table 17: Training time (minutes) and performance on Walker2d across ODE integration steps ( N ), averaged over 5 random seeds. Performance is presented as Mean (Std).
ODE solver
ODE steps
NFE
Time
Perf
Euler
10
10
171
5282.89 (437.57)
Runge–Kutta 2
10
20
219
5519.10 (395.64)
Runge–Kutta 4
10
40
340
5486.24 (184.02)
Runge–Kutta 2
5
10
168
5234.53 (172.47)
Runge–Kutta 4
2
8
164
3988.16 (2367.62)
Appendix
Table 18: Training time (minutes) and performance on Walker2d across ODE solvers, averaged over 5 random seeds. Performance is presented as Mean (Std).
In maximum entropy and behavior-constrained reinforcement learning (RL), the optimal policy is given by a Boltzmann distribution. Generating samples from this distribution can be challenging, as it is often complex and multimodal. While flow policies trained with flow matching provide a powerful framework for representing such distributions, they traditionally rely on access to training data, which is generally unavailable in online RL. To address this limitation, we propose fixed-point flow matching (FFM), a data-free procedure for training and fine-tuning flow policies in RL. FFM leverages the target flow identity to formulate the actor update as a fixed-point iteration, utilizing a flow-matching-style regression objective where the critic's action gradients serve as regression targets. We show that the optimal policy is a fixed point of this iteration. Through extensive experiments, we demonstrate that FFM effectively fine-tunes pretrained flow policies in the offline-to-online setting and enables efficient and accurate training of flow policies from scratch in online RL. These results establish fixed-point flow matching as a principled approach for learning expressive flow-based policies without requiring training data.
Denis Blessing, Onur Celik, Gerhard Neumann
Karlsruhe Institute of Technology (KIT) Autonomous Learning Robots Lab (ALR)
Integrating expressive generative policies, such as flow-matching models, into offline reinforcement learning (RL) allows agents to capture complex, multi-modal behaviors. While Q-learning with Adjoint Matching (QAM) stabilizes policy optimization via the continuous adjoint method, it remains inherently bound to the fixed behavior distribution. This dependence induces a popularity bias that can suppress high-reward actions in low-density regions, and creates a support binding that restricts off-manifold exploration. Existing workarounds, such as appending residual Gaussian policies, often re-introduce the expressivity bottlenecks associated with unimodal distributions. In this work, we propose Maximum Entropy Adjoint Matching (ME-AM), a unified framework that addresses these limitations within the continuous flow formulation. ME-AM incorporates two mechanisms: (1) a Mirror Descent entropy maximization objective that mitigates the popularity bias to facilitate the extraction of optimal policies from offline datasets, and (2) a Mixture Behavior Prior that broadens the geometric support to encompass out-of-distribution high-reward regions. By exploring this extended geometry, ME-AM identifies robust actions while preserving the absolute continuity of the generative vector field. Empirically, ME-AM demonstrates competitive or superior performance compared to prior state-of-the-art (SOTA) methods across a diverse suite of sparse-reward continuous control environments.
Abdelghani Ghanem, Mounir Ghogho
College of Computing Mohammed VI Polytechnic University Rabat 11103, Morocco
We present DeFlow, a decoupled offline RL framework that leverages flow matching to faithfully capture complex behavior manifolds. Optimizing generative policies is computationally prohibitive, typically necessitating backpropagation through ODE solvers. We address this by learning a lightweight refinement module within an explicit, data-derived trust region of the flow manifold, rather than sacrificing the iterative generation capability via single-step distillation. This way, we bypass solver differentiation and eliminate the need for balancing loss terms, ensuring stable improvement while fully preserving the flow's iterative expressivity. Empirically, DeFlow achieves superior performance on the challenging OGBench benchmark and demonstrates efficient offline-to-online adaptation.
Zhancun Mu, Chi Zhang
Peking University, Institute of Artificial Intelligence, China, Beijing.