Compositional Average Velocity Modeling for Efficient Diffusion Offline Reinforcement Learning

Thanh Nguyen; Abu Hanif Muhammad Syarubany; Hwanhee Kim; Chang D. Yoo

cs.LGOct 10, 2026

Compositional Average Velocity Modeling for Efficient Diffusion Offline Reinforcement Learning

Authors: Thanh Nguyen, Abu Hanif Muhammad Syarubany, Hwanhee Kim, Chang D. Yoo

Organizations: The authors are with the School of Electrical Engineering, KAIST, Daejeon, Republic of Korea.

Abstract

Diffusion-based policies have recently emerged as a powerful paradigm for offline reinforcement learning, but their reliance on multi-step denoising introduces substantial computational overhead and requires backpropagation through time during training, limiting scalability and practical deployment. In this work, we propose Instant Flow Q-Learning (InsQL), a single-step generative policy learning framework that retains the expressive capacity of diffusion models while eliminating iterative sampling and BPTT. InsQL builds on the flow-matching perspective of diffusion policies, but replaces marginal velocity parameterization with average-velocity modeling and introduces a principled compositional mechanism, yielding a theoretically grounded objective that learns average velocity consistently across scales. This formulation provides a direct noise-to-action mapping and enables single-step action generation. InsQL further unifies compositional average-velocity modeling with behavior-regularized actor--critic optimization in a single framework, requiring only a single policy network and avoiding distillation, multi-stage procedures, and Jacobian computations during training. At inference time, actions are generated in a single forward pass and can be refined via value-guided candidate selection to improve performance. Extensive experiments on standard offline RL benchmarks and real-world robotic manipulation tasks show that InsQL matches or surpasses state-of-the-art diffusion-based methods while substantially reducing both training and inference time. Overall, these results establish compositional average-velocity modeling as an efficient and scalable alternative to multi-step generative policies for offline RL.

Figures & tables

Fig. 1 : Instant Flow Q-Learning (InsQL) overview. (a) Standard flow-matching (diffusion) policies learn an instantaneous marginal velocity field

v(a_{t},t,s)

and generate actions by multi-step integration. (b) InsQL instead parameterizes the policy with an average-velocity field

u(a_{t},n,t,s)

over finite intervals, enabling single-step action generation by directly mapping noise to action. (c) Velocity composition : long-interval average velocity (blue) is recursively composed from shorter-interval average velocities (green/red); in the infinitesimal limit, this recovers the marginal velocity used in FM. (d) InsQL unifies the one-step policy into a non-BPTT behavior-regularized actor–critic framework, with optional Q-guided candidate selection for test-time refinement.

	Non-Diffusion Policies			Multi-step Diffusion Policies			One-step Flow Policies
Dataset	BC	TD3-BC	IQL	EDP	IDQL	DQL	FQL	InsQL (Ours)
HalfCheetah-Medium-Expert	55.2	90.7	86.7	95.8 ± 0.1	91.3 ± 0.6	95.5 ± 0.1	99.8 ± 0.1	97.5 ± 0.1
Hopper-Medium-Expert	52.5	98.0	91.5	110.8 ± 0.4	110.1 ± 0.7	111.1 ± 0.4	86.2 ± 1.3	109.5 ± 1.3
Walker2d-Medium-Expert	107.5	110.1	109.6	110.4 ± 0.0	110.6 ± 0.0	111.6 ± 0.9	100.5 ± 0.1	112.9 ± 0.1
HalfCheetah-Medium	42.6	48.3	47.4	50.8 ± 0.0	51.5 ± 0.1	52.3 ± 0.2	60.1 ± 0.1	65.0 ± 0.1
Hopper-Medium	52.9	59.3	66.3	72.6 ± 0.2	70.1 ± 2.0	96.5 ± 1.3	74.5 ± 0.2	100.5 ± 1.1

TABLE I : Normalized performance on D4RL benchmarks spanning MuJoCo locomotion and AntMaze navigation. Boldfaced entries denote the best result in each row.

Fig. 2 : Real-World Robotic Manipulation Setup

Fig. 3 : Evaluation tasks. (a–c) MuJoCo locomotion tasks (d) AntMaze Navigation task. (e–f) Real-world robotic manipulation tasks

Method	Steps	Frequency	Training Time
InsQL (-)	1	840.5	7.8
InsQL	1	680.2	7.8
FQL	1	929.6	8.5
DQL	5	238.1	11.7
	10	150.1	16.1
	20	75.2	24.8

TABLE II: Action frequency (Hz) and training wall-clock time (hours) for

10^{6}

training steps, averaged over MuJoCo tasks, across methods and sampling steps. InsQL (-) denotes InsQL without Q-guided sampling.

Method	Sweep Cube	Pickup Banana
BC	0.0	0.0
DQL	0.6	0.9
InsQL	0.7	1.0

TABLE III : Real-robot success rates. Success rate on Sweep Cube and Pickup Banana for BC, DQL, and InsQL (higher is better).

Fig. 4 : FM vs. InsQL on a multimodal checkerboard. FM requires multi-step generation to cover the modes and collapses under few-step evaluation, whereas InsQL achieves near-complete mode coverage and closely matches the target distribution in a single step.

$\lambda$	1	0.75	0.5	0.25	0
Medium Expert	79.8	95.5	97.5	93.1	1.5
Medium	43.3	63.8	65.0	61.5	-1.5
Medium Replay	48.0	49.7	52.1	51.3	9.6

TABLE IV: Ablation study on the boundary conditioning ratio

\lambda

for HalfCheetah tasks under different dataset regimes. Results are reported as D4RL normalized scores.

Fig. 5 : Ablation on the effect of Q-guided sampling on HalfCheetah.

K

denotes the number of candidate actions sampled per state; we choose the action with the highest predicted

Q

Offline RL

Offline reinforcement learning learns decision-making policies from fixed datasets without interacting with the environment during training.

See all 304 papers

Fast and Highly Expressive Policy Learning for Offline Reinforcement Learning via Bootstrapped Flow Q-Learning
Jun 9, 2026Thanh Nguyen, Tri Ton, Hongbin Choe +2Flow Matching Q-Learning
Department of Electrical Engineering, KAIST, Daejeon, Korea.
Mean Flow Policy Optimization
Apr 16, 2026Xiaoyi Dong, Xi Sheryl Zhang, Jian ChengMeanFlow Mean Flow
C2DL, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · AiRiA +1
Score-Based One-step MeanFlow Policy Optimization
May 22, 2026Kyungyoon Kim, Donghyeon Ki, Hee-Jun Ahn +1Flow Matching MeanFlow
Korea University, Decision Making Lab · Gauss Labs Inc.

Jun 9, 2026J/K move · Enter open · S save

Fast and Highly Expressive Policy Learning for Offline Reinforcement Learning via Bootstrapped Flow Q-Learning

Thanh Nguyen, Tri Ton, Hongbin Choe, Tung M. Luu+1

Department of Electrical Engineering, KAIST, Daejeon, Korea.

Flow Matching Q-Learning Offline RL Flow-Based Policy Learning

Open paper AskPDF arXiv

Diffusion-based Q-learning has emerged as a powerful paradigm for offline reinforcement learning, but its reliance on multi-step denoising makes both training and inference computationally expensive and brittle. Recent efforts to accelerate diffusion Q-learning toward single-step action generation typically introduce auxiliary networks, policy distillation, or multi-phase training, which frequently compromise simplicity, stability, or performance. To address these limitations, we introduce Bootstrapped Flow Q-Learning (BFQ), a novel framework that enables accurate single-step action generation during both training and inference, without auxiliary networks or distillation procedures. BFQ adopts a divide-and-conquer view of the displacement vector along the flow path: it begins by learning short-range displacements that can be accurately estimated from the Flow Matching marginal velocity, and bootstraps these components to directly learn a noise-to-action mapping in a single step. This formulation eliminates multi-step denoising, resulting in a learning procedure that is substantially faster, simpler, and more robust. Extensive D4RL evaluations show that BFQ improves performance while significantly reducing computational cost compared to multi-step diffusion baselines, demonstrating that single-step action generation suffices for high-performance offline Reinforcement Learning.

Abstract

Offline RL

Explore similar work

Fast and Highly Expressive Policy Learning for Offline Reinforcement Learning via Bootstrapped Flow Q-Learning

Mean Flow Policy Optimization

Score-Based One-step MeanFlow Policy Optimization