Offline reinforcement learning aims to learn effective policies from fixed datasets without environment interaction. Although generative models such as diffusion and flow matching enhance policy expressiveness, they typically start from uninformative Gaussian noise and generate actions through multi-step iteration, leading to high computational costs and failing to effectively leverage suboptimal trajectories in datasets. We propose MeanFlow2Better (MF2B), which reformulates offline RL as generative decision chunk optimization from current performance status to better performance status in a high-level latent decision space. MF2B maps trajectory chunks to high-level latent decision representations through an encoder, placing generation starting points within a structured decision space sculpted by behavioral data. The model learns a shared conditional mean velocity field in this space, constructing two flow mappings: from Gaussian-Induced Decision Sources toward positive behavioral decision regions, and from suboptimal decisions toward high-value decision regions. Both flow mappings complete decision refinement through one-step ODE integration over the unit time interval Δt=1, requiring neither multi-step iterative sampling nor backpropagation through time. Furthermore, we propose hierarchical chunk-level advantage to guide the flow direction toward high-return regions, where the high-level decisions cover multiple low-level actions, leading to sequence length compression from T to K≈T/H and significantly lower computational overhead. On the D4RL Gym-MuJoCo benchmark, MF2B achieves an average score of 90.91, matching or surpassing strong baselines.
Figure 1: High-level latent decision space analysis. Left: t-SNE visualization of Ztop (green), Zlow (blue), Zpos (yellow), and Zneg (red). Right: KDE plots of the first three latent dimensions comparing the four distributions.
Dataset
H=5, Kmax =200
H=10, Kmax =100
H=20, Kmax =50
HalfCheetah MR
43.03 ± 0.35
43.81 ± 0.30
42.95 ± 0.70
Hopper MR
100.03 ± 1.39
104.36 ± 2.12
99.41 ± 2.62
Walker2d MR
91.73 ± 1.23
96.24 ± 1.02
93.28 ± 1.85
Table 3: Results for Different Chunk Lengths and Maximum High-Level Chunk Counts
Dataset
Kmax =25
Kmax =50
Kmax =100
HalfCheetah MR
11.74 ± 0.14
22.07 ± 0.22
43.81 ± 0.30
Hopper MR
25.69 ± 0.43
52.02 ± 0.45
104.36 ± 2.12
Walker2d MR
21.13 ± 0.86
42.58 ± 0.29
96.24 ± 1.02
Table 4: Results for Different Kmax with Fixed H=10
Dataset
ρ =0.1
ρ =0.2
ρ =0.5
ρ =0.8
HalfCheetah MR
42.32 ± 0.36
43.81 ± 0.30
43.00 ± 0.51
42.66 ± 0.30
Hopper MR
101.45 ± 2.33
104.36 ± 2.12
99.49 ± 3.48
66.24 ± 5.75
Walker2d MR
96.77 ± 2.34
96.24 ± 1.02
95.70 ± 2.17
85.92 ± 8.10
Table 5: Results for Different High-Value Timestep Coverage Ratios
Diffusion models have recently emerged as expressive policy representations for online reinforcement learning (RL). However, their iterative generative processes introduce substantial training and inference overhead. To overcome this limitation, we propose to represent policies using MeanFlow models, a class of few-step flow-based generative models, to improve training and inference efficiency over diffusion-based RL approaches. To promote exploration, we optimize MeanFlow policies under the maximum entropy RL framework via soft policy iteration, and address two key challenges specific to MeanFlow policies: action likelihood evaluation and soft policy improvement. Experiments on MuJoCo, DeepMind Control Suite and HumanoidBench benchmarks demonstrate that our method, Mean Flow Policy Optimization (MFPO), achieves performance comparable to or exceeding current diffusion-based baselines while considerably reducing training and inference time. Our code is available at https://github.com/dongxiaoyi-xyz/MFPO.
Xiaoyi Dong, Xi Sheryl Zhang, Jian Cheng
C2DL, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · AiRiA +1
Generative policies based on diffusion models and flow matching have shown strong promise for offline reinforcement learning (RL), but their applicability remains largely confined to continuous action spaces. To address a broader range of offline RL settings, we extend flow matching to a general framework that supports discrete action spaces with multiple objectives. Specifically, we replace continuous flows with continuous-time Markov chains, trained using a Q-weighted flow matching objective. We then extend our design to multi-agent settings, mitigating the exponential growth of joint action spaces via a factorized conditional path. We theoretically show that, under idealized conditions, optimizing this objective recovers the optimal policy. Extensive experiments further demonstrate that our method performs robustly across diverse settings and benchmarks, including high-dimensional control, multi-agent games, and dynamically changing preferences over multiple objectives, while outperforming traditional offline RL methods in practical multi-modal decision-making scenarios. Our discrete framework can also be applied to continuous-control problems through action quantization, providing a flexible trade-off between representational complexity and performance.
Fairoz Nower Khan, Nabuat Zaman Nahim, Ruiquan Huang +2
Department of Computer Science, University of Kentucky · Department of Computing and Information Sciences, Rochester Institute of Technology
The dataset distributions in offline reinforcement learning (RL) often exhibit complex and multi-modal distributions, necessitating expressive policies to capture such distributions beyond widely-used Gaussian policies. To handle such complex and multi-modal datasets, in this paper, we propose Flow Actor-Critic, a new actor-critic method for offline RL, based on recent flow policies. The proposed method not only uses the flow model for actor as in previous flow policies but also exploits the expressive flow model for conservative critic acquisition to prevent Q-value explosion in out-of-data regions. To this end, we propose a new form of critic regularizer based on the flow behavior proxy model obtained as a byproduct of flow-based actor design. Leveraging the flow model in this joint way, we achieve new state-of-the-art performance for test datasets of offline RL including the D4RL and recent OGBench benchmarks.