Diffusion-based policies have recently emerged as a powerful paradigm for offline reinforcement learning, but their reliance on multi-step denoising introduces substantial computational overhead and requires backpropagation through time during training, limiting scalability and practical deployment. In this work, we propose Instant Flow Q-Learning (InsQL), a single-step generative policy learning framework that retains the expressive capacity of diffusion models while eliminating iterative sampling and BPTT. InsQL builds on the flow-matching perspective of diffusion policies, but replaces marginal velocity parameterization with average-velocity modeling and introduces a principled compositional mechanism, yielding a theoretically grounded objective that learns average velocity consistently across scales. This formulation provides a direct noise-to-action mapping and enables single-step action generation. InsQL further unifies compositional average-velocity modeling with behavior-regularized actor--critic optimization in a single framework, requiring only a single policy network and avoiding distillation, multi-stage procedures, and Jacobian computations during training. At inference time, actions are generated in a single forward pass and can be refined via value-guided candidate selection to improve performance. Extensive experiments on standard offline RL benchmarks and real-world robotic manipulation tasks show that InsQL matches or surpasses state-of-the-art diffusion-based methods while substantially reducing both training and inference time. Overall, these results establish compositional average-velocity modeling as an efficient and scalable alternative to multi-step generative policies for offline RL.
Figures & tables
Fig. 1 : Instant Flow Q-Learning (InsQL) overview. (a) Standard flow-matching (diffusion) policies learn an instantaneous marginal velocity field v(at,t,s) and generate actions by multi-step integration. (b) InsQL instead parameterizes the policy with an average-velocity field u(at,n,t,s) over finite intervals, enabling single-step action generation by directly mapping noise to action. (c) Velocity composition : long-interval average velocity (blue) is recursively composed from shorter-interval average velocities (green/red); in the infinitesimal limit, this recovers the marginal velocity used in FM. (d) InsQL unifies the one-step policy into a non-BPTT behavior-regularized actor–critic framework, with optional Q-guided candidate selection for test-time refinement.
Non-Diffusion Policies
Multi-step Diffusion Policies
One-step Flow Policies
Dataset
BC
TD3-BC
IQL
EDP
IDQL
DQL
FQL
InsQL (Ours)
HalfCheetah-Medium-Expert
55.2
90.7
86.7
95.8 ± 0.1
91.3 ± 0.6
95.5 ± 0.1
99.8 ± 0.1
97.5 ± 0.1
Hopper-Medium-Expert
52.5
98.0
91.5
110.8 ± 0.4
110.1 ± 0.7
111.1 ± 0.4
86.2 ± 1.3
109.5 ± 1.3
Walker2d-Medium-Expert
107.5
110.1
109.6
110.4 ± 0.0
110.6 ± 0.0
111.6 ± 0.9
100.5 ± 0.1
112.9 ± 0.1
HalfCheetah-Medium
42.6
48.3
47.4
50.8 ± 0.0
51.5 ± 0.1
52.3 ± 0.2
60.1 ± 0.1
65.0 ± 0.1
Hopper-Medium
52.9
59.3
66.3
72.6 ± 0.2
70.1 ± 2.0
96.5 ± 1.3
74.5 ± 0.2
100.5 ± 1.1
TABLE I : Normalized performance on D4RL benchmarks spanning MuJoCo locomotion and AntMaze navigation. Boldfaced entries denote the best result in each row.
TABLE II: Action frequency (Hz) and training wall-clock time (hours) for 106 training steps, averaged over MuJoCo tasks, across methods and sampling steps. InsQL (-) denotes InsQL without Q-guided sampling.
Method
Sweep Cube
Pickup Banana
BC
0.0
0.0
DQL
0.6
0.9
InsQL
0.7
1.0
TABLE III : Real-robot success rates. Success rate on Sweep Cube and Pickup Banana for BC, DQL, and InsQL (higher is better).
Fig. 4 : FM vs. InsQL on a multimodal checkerboard. FM requires multi-step generation to cover the modes and collapses under few-step evaluation, whereas InsQL achieves near-complete mode coverage and closely matches the target distribution in a single step.
λ
1
0.75
0.5
0.25
0
Medium Expert
79.8
95.5
97.5
93.1
1.5
Medium
43.3
63.8
65.0
61.5
-1.5
Medium Replay
48.0
49.7
52.1
51.3
9.6
TABLE IV: Ablation study on the boundary conditioning ratio λ for HalfCheetah tasks under different dataset regimes. Results are reported as D4RL normalized scores.
Fig. 5 : Ablation on the effect of Q-guided sampling on HalfCheetah. K denotes the number of candidate actions sampled per state; we choose the action with the highest predicted Q .
Diffusion-based Q-learning has emerged as a powerful paradigm for offline reinforcement learning, but its reliance on multi-step denoising makes both training and inference computationally expensive and brittle. Recent efforts to accelerate diffusion Q-learning toward single-step action generation typically introduce auxiliary networks, policy distillation, or multi-phase training, which frequently compromise simplicity, stability, or performance. To address these limitations, we introduce Bootstrapped Flow Q-Learning (BFQ), a novel framework that enables accurate single-step action generation during both training and inference, without auxiliary networks or distillation procedures. BFQ adopts a divide-and-conquer view of the displacement vector along the flow path: it begins by learning short-range displacements that can be accurately estimated from the Flow Matching marginal velocity, and bootstraps these components to directly learn a noise-to-action mapping in a single step. This formulation eliminates multi-step denoising, resulting in a learning procedure that is substantially faster, simpler, and more robust. Extensive D4RL evaluations show that BFQ improves performance while significantly reducing computational cost compared to multi-step diffusion baselines, demonstrating that single-step action generation suffices for high-performance offline Reinforcement Learning.
Thanh Nguyen, Tri Ton, Hongbin Choe +2
Department of Electrical Engineering, KAIST, Daejeon, Korea.
Diffusion models have recently emerged as expressive policy representations for online reinforcement learning (RL). However, their iterative generative processes introduce substantial training and inference overhead. To overcome this limitation, we propose to represent policies using MeanFlow models, a class of few-step flow-based generative models, to improve training and inference efficiency over diffusion-based RL approaches. To promote exploration, we optimize MeanFlow policies under the maximum entropy RL framework via soft policy iteration, and address two key challenges specific to MeanFlow policies: action likelihood evaluation and soft policy improvement. Experiments on MuJoCo, DeepMind Control Suite and HumanoidBench benchmarks demonstrate that our method, Mean Flow Policy Optimization (MFPO), achieves performance comparable to or exceeding current diffusion-based baselines while considerably reducing training and inference time. Our code is available at https://github.com/dongxiaoyi-xyz/MFPO.
Xiaoyi Dong, Xi Sheryl Zhang, Jian Cheng
C2DL, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · AiRiA +1
Diffusion and flow matching have emerged as expressive policy classes in reinforcement learning, but their reliance on multi-step denoising imposes substantial computational overhead at inference time, which is particularly problematic in online RL. MeanFlow offers a promising alternative by learning an average velocity field that maps noise to data in a single network evaluation. However, MeanFlow typically requires samples from the target distribution to construct its target velocity field, which are unavailable in online RL. We propose Score-Based One-step MeanFlow Policy Optimization (SOM), an actor-critic algorithm that resolves this by constructing the target velocity field directly from the Q-function via score estimation and a probability flow ODE, thereby concentrating probability mass on high-value modes. In the fully online RL setting, SOM achieves state-of-the-art performance on locomotion tasks with a single generation step, while substantially reducing both training and inference time compared to prior diffusion- and flow-matching-based policies.
Kyungyoon Kim, Donghyeon Ki, Hee-Jun Ahn +1
Korea University, Decision Making Lab · Gauss Labs Inc.