DeFlow: Decoupling Behavior-Prior Modeling and Value Maximization for Offline Policy Extraction

Zhancun Mu; Chi Zhang

cs.LGJan 15, 2026

DeFlow: Decoupling Behavior-Prior Modeling and Value Maximization for Offline Policy Extraction

Authors: Zhancun Mu, Chi Zhang

Organizations: Peking University, Institute of Artificial Intelligence, China, Beijing.

Abstract

We present DeFlow, a decoupled offline RL framework that leverages flow matching to faithfully capture complex behavior manifolds. Optimizing generative policies is computationally prohibitive, typically necessitating backpropagation through ODE solvers. We address this by learning a lightweight refinement module within an explicit, data-derived trust region of the flow manifold, rather than sacrificing the iterative generation capability via single-step distillation. This way, we bypass solver differentiation and eliminate the need for balancing loss terms, ensuring stable improvement while fully preserving the flow's iterative expressivity. Empirically, DeFlow achieves superior performance on the challenging OGBench benchmark and demonstrates efficient offline-to-online adaptation.

Figures & tables

Figure 1: DeFlow Framework. (a) Iterative Flow Prior: A multi-step flow policy models multimodal behavior. (b) Shortcut Extraction: One-step distillation enables direct Q-optimization but may compress the generated action distribution. (c) DeFlow: We keep the iterative prior (Blue) and train a lightweight residual module (Orange) to perform local value improvement while penalizing large deviations from the prior.

	Gaussian Policies			Flow / Generative Extraction
Task Category	BC	IQL	ReBRAC	FAWAC	FBRAC	IFQL	SORL	FQL	DeFlow
antmaze-large (5)	$11$ $\pm 1$	$53$ $\pm 3$	$81$ $\pm 5$	$6$ $\pm 1$	$60$ $\pm 6$	$28$ $\pm 5$	$\mathbf{89}$ $\pm 2$	$79$ $\pm 3$	$81$ $\pm 3$
antmaze-giant (5)	$0$ $\pm 0$	$4$ $\pm 1$	$\mathbf{26}$ $\pm 8$	$0$ $\pm 0$	$4$ $\pm 4$	$3$ $\pm 2$	$9$ $\pm 6$	$9$ $\pm 6$	$12$ $\pm 5$
humanoidmaze-medium (5)	$2$ $\pm 1$	$33$ $\pm 2$	$22$ $\pm 8$	$19$ $\pm 1$	$38$ $\pm 5$	$60$ $\pm 14$	$\mathbf{64}$ $\pm 4$	$58$ $\pm 5$	$48$ $\pm 4$
humanoidmaze-large (5)	$1$ $\pm 0$	$2$ $\pm 1$	$2$ $\pm 1$	$0$ $\pm 0$	$2$ $\pm 0$	$\mathbf{11}$ $\pm 2$	$5$ $\pm 2$	$4$ $\pm 2$	$5$ $\pm 2$
antsoccer-arena (5)	$1$ $\pm 0$	$8$ $\pm 2$	$0$ $\pm 0$	$12$ $\pm 0$	$16$ $\pm 1$	$33$ $\pm 6$	$\mathbf{69}$ $\pm 2$	$60$ $\pm 2$	$\mathbf{67}$ $\pm 3$

Table 1: Offline RL results on OGBench and D4RL. We report DeFlow against Gaussian-policy and flow-based/generative policy-extraction baselines. SORL results are included where aligned OGBench numbers are available; missing entries are marked by “-” and are not used when identifying row-wise best scores. Results are averaged over 8 seeds for state tasks and 4 seeds for visual tasks when applicable. Bold entries are at least

0.95\times

the best reported score in their row. See Table 16 for the full task-level version of the non-SORL baselines. Counts in parentheses indicate the number of tasks.

Task	IQL	ReBRAC	Cal-QL	RLPD	IFQL	FQL	DeFlow
humanoidmaze-medium	$21$ $\pm 13$ $\to$ $16$ $\pm 8$	$16$ $\pm 20$ $\to$ $1$ $\pm 1$	$0$ $\pm 0$ $\to$ $0$ $\pm 0$	$0$ $\pm 0$ $\to$ $8$ $\pm 10$	$56$ $\pm 35$ $\to$ $\mathbf{82}$ $\pm 20$	$12$ $\pm 7$ $\to$ $22$ $\pm 12$	$13$ $\pm 5$ $\to$ $65$ $\pm 12$
antsoccer-arena	$2$ $\pm 1$ $\to$ $0$ $\pm 0$	$0$ $\pm 0$ $\to$ $0$ $\pm 0$	$0$ $\pm 0$ $\to$ $0$ $\pm 0$	$0$ $\pm 0$ $\to$ $0$ $\pm 0$	$26$ $\pm 15$ $\to$ $39$ $\pm 10$	$28$ $\pm 8$ $\to$ $\mathbf{86}$ $\pm 5$	$44$ $\pm 8$ $\to$ $\mathbf{86}$ $\pm 3$
cube-double	$0$ $\pm 1$ $\to$ $0$ $\pm 0$	$6$ $\pm 5$ $\to$ $28$ $\pm 28$	$0$ $\pm 0$ $\to$ $0$ $\pm 0$	$0$ $\pm 0$ $\to$ $0$ $\pm 0$	$12$ $\pm 9$ $\to$ $40$ $\pm 5$	$40$ $\pm 11$ $\to$ $\mathbf{92}$ $\pm 3$	$48$ $\pm 13$ $\to$ $\mathbf{93}$ $\pm 4$
scene	$14$ $\pm 11$ $\to$ $10$ $\pm 9$	$55$ $\pm 10$ $\to$ $\mathbf{100}$ $\pm 0$	$1$ $\pm 2$ $\to$ $50$ $\pm 53$	$0$ $\pm 0$ $\to$ $\mathbf{100}$ $\pm 0$	$0$ $\pm 1$ $\to$ $60$ $\pm 39$	$82$ $\pm 11$ $\to$ $\mathbf{100}$ $\pm 1$	$58$ $\pm 14$ $\to$ $\mathbf{100}$ $\pm 1$
puzzle-4x4	$5$ $\pm 2$ $\to$ $1$ $\pm 1$	$8$ $\pm 4$ $\to$ $14$ $\pm 35$	$0$ $\pm 0$ $\to$ $0$ $\pm 0$	$0$ $\pm 0$ $\to$ $\mathbf{100}$ $\pm 1$	$23$ $\pm 6$ $\to$ $19$ $\pm 33$	$8$ $\pm 3$ $\to$ $38$ $\pm 52$	$4$ $\pm 4$ $\to$ $\mathbf{100}$ $\pm 0$
antmaze-umaze-v2	$77$ $\to$ $\mathbf{96}$	$98$ $\to$ $75$	$77$ $\to$ $\mathbf{100}$	$0$ $\pm 0$ $\to$ $\mathbf{98}$ $\pm 3$	$94$ $\pm 5$ $\to$ $\mathbf{96}$ $\pm 2$	$97$ $\pm 2$ $\to$ $\mathbf{99}$ $\pm 1$	$96$ $\pm 3$ $\to$ $\mathbf{99}$ $\pm 2$

Table 2: Offline-to-online RL results. The results are averaged over

8

seeds unless otherwise mentioned. Entries show offline

\to

online performance; OGBench rows use the default single-task environment.

Figure 2: Visualization of Action Landscapes. Overlay of Q-value contours and action samples. Gray : Base Flow samples (data proxy); Blue : FQL (one-step); Red : DeFlow (ours). Arrows indicate the refinement trajectory. DeFlow performs local residual updates around base-flow proposals, illustrating smaller local shifts in this representative state.

Figure 3: Offline-to-online RL results ( $\mathbf{8}$ seeds).

Family	Update path	Eval. budget	Proposal use
Iterative BPTT	solver + critic	$MC_{v}$	update generator
Shortcut / FQL	shortcut + critic	1–few steps	compress actor
SORL-style	shortcut + critic	$k$ steps; optional ranking	scale inference
DeFlow	residual + critic	$MC_{v}+C_{f}$	local residual

Table 3: Compute trade-offs for generative policy extraction.

M

is the number of flow-solver steps;

C_{v}

C_{f}

, and

C_{Q}

denote vector-field, residual-module, and critic costs.

Method	Offline train	Online train	Inference
FQL	$2.2$ $\pm 1.0$	$5.1$ $\pm 0.5$	$4.2$ $\pm 0.7$
DeFlow	$2.4$ $\pm 1.0$	$5.3$ $\pm 0.4$	$4.5$ $\pm 0.5$
DeFlow (frozen prior)	–	$5.2$ $\pm 0.5$	–

Table 4: Training and inference time comparison in milliseconds per step.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

	cube-double	antmaze-large	puzzle-3x3	antsoccer
IAV	$0.013$	$0.018$	$0.007$	$0.026$

Appendix

Table 8

Quantity	Main benchmark protocol	Additional analysis / budget
Residual budget $\delta$	Task-dependent IAV-guided choice; retained for O2O	$\{0.0005,0.001,0.005,0.01,0.02\}$ diagnostic in Table 6
Penalty $\alpha$	Adapted in log space; not a fixed-coefficient grid	Original nominal fixed settings $\{0.03,0.1,0.3,1,3\}$ in Table 9
Residual architecture	Four hidden layers, width 512; no per-task depth tuning	One/two layers, width 512; four seeds per setting; Section C.4
Prior and critic	FQL-based configuration; prior learns by flow matching	Held at the same architecture within each capacity comparison
Inference budget	Multi-step proposal followed by one residual	Ten Euler steps in capacity runs; no candidate ranking
Evaluation selection	Fixed-step main evaluation	Final scheduled checkpoint for capacity; 20 or 50 episodes per seed

Appendix

Table 5: Hyperparameter choices and diagnostic scope. The sensitivity grids are diagnostic settings, not a claimed search history for selecting the main results.

$\delta$	$0.0005$	$0.001$	$0.005$	$0.01$	$0.02$
Score	$55$ $\pm 8$	$50$ $\pm 7$	$27$ $\pm 4$	$8$ $\pm 5$	$4$ $\pm 2$

Appendix

Table 6: Sensitivity to the residual budget

\delta

on a representative manipulation diagnostic. Larger budgets permit larger residual moves and can quickly degrade performance.

Diagnostic	Setting	Baseline	DeFlow / Variant
Mixed human/expert data	score	FQL: $21$ $\pm 3$	DeFlow: $37$ $\pm 5$
Refinement module	full action-conditioned $f_{\phi}(s,a_{b})$	-	$40$ $\pm 5$
Refinement module	remove action conditioning	-	$5$ $\pm 3$

Appendix

Table 7: Diagnostic checks for prior coverage and action-conditioned refinement. The mixed dataset uses 50% human and 50% expert data; the refinement ablation removes action conditioning from

f_{\phi}(s,a_{b})

Task	FQL	floq	DeFlow	DeFlow+floq
cube-double-play	$25$ $\pm 6$	$47$ $\pm 15$	$40$ $\pm 5$	$\mathbf{58}$ $\pm 5$
puzzle-3x3-play	$29$ $\pm 5$	$37$ $\pm 7$	$43$ $\pm 4$	$\mathbf{50}$ $\pm 3$

Appendix

Table 8: Success rates on manipulation tasks when combining DeFlow with floq’s value function.

Task	Phase	$c=0.03$	$0.1$	$0.3$	$1$	$3$	Adaptive
cube-double	Offline	$0\pm 0$	$0\pm 0$	$0\pm 0$	$44\pm 2$	$7\pm 2$	$48\pm 13$
	Online	$0\pm 0$	$0\pm 0$	$0\pm 0$	$10\pm 3$	$89\pm 5$	$93\pm 4$
puzzle-3x3	Offline	$0\pm 0$	$0\pm 0$	$7\pm 7$	$0\pm 0$	$35\pm 8$	$23\pm 8$
	Online	$100\pm 1$	$100\pm 0$	$100\pm 0$	$0\pm 0$	$0\pm 0$	$100\pm 0$

Appendix

Table 9: Original fixed-penalty sweep versus adaptation, separated by training phase. The fixed columns retain the nominal sweep setting

c

; see the implementation qualification below.

Task	Base flow only	Base flow + residual
cube-double task1	$10\pm 2$	$62\pm 12$
puzzle-3x3 task2	$0.5\pm 0.9$	$35\pm 11$

Appendix

Table 10: Inference-time residual attribution at the same trained checkpoints. These are separate evaluations, not replacements for the main benchmark scores.

Task	One layer	Two layers	Four layers
cube-double task1	$81\pm 9$	$74\pm 23$	$69\pm 10\,\dagger$
puzzle-3x3 task2	$6\pm 3$	$55\pm 7$	$38\pm 11\,*$
scene task2	$52\pm 12$	$70\pm 13$	$59\pm 17\,*$

Appendix

Table 11: Residual-depth diagnostics. All hidden layers have width 512. Cube’s four-layer entry is a matched four-seed rerun (

\dagger

); puzzle and scene four-layer entries are the original eight-seed benchmark references (

*

), not matched reruns.

Task	One layer	Two layers
antmaze-large task2	$48\pm 14$	$73\pm 6$
humanoidmaze-medium task2	$43\pm 18$	$73\pm 10$
antsoccer-arena task2	$28\pm 12$	$76\pm 3$
cube-single task2	$98\pm 1$	$97\pm 3$
puzzle-4x4 task1	$22\pm 2$	$24\pm 7$
pen-cloned	$63\pm 20$	$64\pm 8$

Appendix

Table 12: Broader one- versus two-layer sensitivity diagnostic. OGBench entries are success percentages; pen-cloned uses D4RL normalized return. No best-depth selection is applied to the main benchmark.

Task	$\delta$	Discount	Target aggregation
cube-double task1	$0.001$	$0.99$	mean
puzzle-3x3 task2	$0.0005$	$0.99$	mean
scene task2	$0.001$	$0.99$	mean
antmaze-large task2	$0.01$	$0.99$	min
humanoidmaze-medium task2	$0.001$	$0.995$	mean
antsoccer-arena task2	$0.01$	$0.995$	mean

Appendix

Table 13: Task-specific settings for the new capacity diagnostics. Target aggregation refers to the Bellman target, not candidate ranking.

	IQL	ReBRAC	IDQL	SRPO	CAC	FAWAC	FBRAC	IFQL	FQL	DeFlow
Task	$\alpha$	$(\alpha_{1},\alpha_{2})$	$N$	$\beta$	$\eta$	$\alpha$	$\alpha$	$N$	$\alpha$	$\delta$
antmaze-large-navigate-singletask-task1-v0	$10$	$(0.003,0.01)$	$32$	$0.3$	$1$	$3$	$3$	$32$	$10$	$0.01$
antmaze-large-navigate-singletask-task2-v0	$10$	$(0.003,0.01)$	$32$	$0.3$	$1$	$3$	$3$	$32$	$10$	$0.01$
antmaze-large-navigate-singletask-task3-v0	$10$	$(0.003,0.01)$	$32$	$0.3$	$1$	$3$	$3$	$32$	$10$	$0.01$
antmaze-large-navigate-singletask-task4-v0	$10$	$(0.003,0.01)$	$32$	$0.3$	$1$	$3$	$3$	$32$	$10$	$0.01$
antmaze-large-navigate-singletask-task5-v0	$10$	$(0.003,0.01)$	$32$	$0.3$	$1$	$3$	$3$	$32$	$10$	$0.01$

Appendix

Table 14: Task-specific hyperparameters for offline RL. We take hyperparameters from FQL [ 24 ] for comparison.

	IQL	ReBRAC	IDQL	SRPO	CAC	FAWAC	FBRAC	IFQL	FQL	DeFlow
Task	$\alpha$	$(\alpha_{1},\alpha_{2})$	$N$	$\beta$	$\eta$	$\alpha$	$\alpha$	$N$	$\alpha$	$\delta$
cube-double-play-singletask-task1-v0	$0.3$	$(0.1,0)$	$32$	$0.1$	$0.3$	$0.3$	$100$	$32$	$300$	$0.001$
cube-double-play-singletask-task2-v0	$0.3$	$(0.1,0)$	$32$	$0.1$	$0.3$	$0.3$	$100$	$32$	$300$	$0.001$
cube-double-play-singletask-task3-v0	$0.3$	$(0.1,0)$	$32$	$0.1$	$0.3$	$0.3$	$100$	$32$	$300$	$0.001$
cube-double-play-singletask-task4-v0	$0.3$	$(0.1,0)$	$32$	$0.1$	$0.3$	$0.3$	$100$	$32$	$300$	$0.001$
cube-double-play-singletask-task5-v0	$0.3$	$(0.1,0)$	$32$	$0.1$	$0.3$	$0.3$	$100$	$32$	$300$	$0.001$

Appendix

Table 15: Task-specific hyperparameters for offline RL (continued).

	Gaussian Policies			Diffusion Policies			Flow Policies
Task	BC	IQL	ReBRAC	IDQL	SRPO	CAC	FAWAC	FBRAC	IFQL	FQL	Deflow
antmaze-large-navigate-singletask-task1-v0 (*)	$0$ $\pm 0$	$48$ $\pm 9$	$\mathbf{91}$ $\pm 10$	$0$ $\pm 0$	$0$ $\pm 0$	$42$ $\pm 7$	$1$ $\pm 1$	$70$ $\pm 20$	$24$ $\pm 17$	$80$ $\pm 8$	$\mathbf{88}$ $\pm 6$
antmaze-large-navigate-singletask-task2-v0	$6$ $\pm 3$	$42$ $\pm 6$	$\mathbf{88}$ $\pm 4$	$14$ $\pm 8$	$4$ $\pm 4$	$1$ $\pm 1$	$0$ $\pm 1$	$35$ $\pm 12$	$8$ $\pm 3$	$57$ $\pm 10$	$71$ $\pm 10$
antmaze-large-navigate-singletask-task3-v0	$29$ $\pm 5$	$72$ $\pm 7$	$51$ $\pm 18$	$26$ $\pm 8$	$3$ $\pm 2$	$49$ $\pm 10$	$12$ $\pm 4$	$83$ $\pm 15$	$52$ $\pm 17$	$\mathbf{93}$ $\pm 3$	$\mathbf{88}$ $\pm 5$
antmaze-large-navigate-singletask-task4-v0	$8$ $\pm 3$	$51$ $\pm 9$	$\mathbf{84}$ $\pm 7$	$62$ $\pm 25$	$45$ $\pm 19$	$17$ $\pm 6$	$10$ $\pm 3$	$37$ $\pm 18$	$18$ $\pm 8$	$\mathbf{80}$ $\pm 4$	$77$ $\pm 5$
antmaze-large-navigate-singletask-task5-v0	$10$ $\pm 3$	$54$ $\pm 22$	$\mathbf{90}$ $\pm 2$	$2$ $\pm 2$	$1$ $\pm 1$	$55$ $\pm 6$	$9$ $\pm 5$	$76$ $\pm 8$	$38$ $\pm 18$	$83$ $\pm 4$	$84$ $\pm 3$

Appendix

Table 16: Full offline RL results. We present the full results on the

73

OGBench and D4RL tasks. (*) indicates the default task in each environment. The results are averaged over

8

seeds (

4

seeds for pixel-based tasks) unless otherwise mentioned.

	Gaussian Policies			Diffusion Policies			Flow Policies
Task	BC	IQL	ReBRAC	IDQL	SRPO	CAC	FAWAC	FBRAC	IFQL	FQL	Deflow
cube-double-play-singletask-task1-v0	$8$ $\pm 3$	$27$ $\pm 5$	$45$ $\pm 6$	$39$ $\pm 19$	$7$ $\pm 6$	$21$ $\pm 8$	$21$ $\pm 7$	$47$ $\pm 11$	$35$ $\pm 9$	$\mathbf{61}$ $\pm 9$	$\mathbf{65}$ $\pm 6$
cube-double-play-singletask-task2-v0 (*)	$0$ $\pm 0$	$1$ $\pm 1$	$7$ $\pm 3$	$16$ $\pm 10$	$0$ $\pm 0$	$2$ $\pm 2$	$2$ $\pm 1$	$22$ $\pm 12$	$9$ $\pm 5$	$36$ $\pm 6$	$\mathbf{49}$ $\pm 13$
cube-double-play-singletask-task3-v0	$0$ $\pm 0$	$0$ $\pm 0$	$4$ $\pm 1$	$17$ $\pm 8$	$0$ $\pm 1$	$3$ $\pm 1$	$1$ $\pm 1$	$4$ $\pm 2$	$8$ $\pm 5$	$22$ $\pm 5$	$\mathbf{34}$ $\pm 15$
cube-double-play-singletask-task4-v0	$0$ $\pm 0$	$0$ $\pm 0$	$1$ $\pm 1$	$0$ $\pm 1$	$0$ $\pm 0$	$0$ $\pm 1$	$0$ $\pm 0$	$0$ $\pm 1$	$1$ $\pm 1$	$5$ $\pm 2$	$\mathbf{7}$ $\pm 4$
cube-double-play-singletask-task5-v0	$0$ $\pm 0$	$4$ $\pm 3$	$4$ $\pm 2$	$1$ $\pm 1$	$0$ $\pm 0$	$3$ $\pm 2$	$2$ $\pm 1$	$2$ $\pm 2$	$17$ $\pm 6$	$19$ $\pm 10$	$\mathbf{46}$ $\pm 11$

Appendix

Table 17: Full offline RL results (continued).

Offline RL

Offline reinforcement learning learns decision-making policies from fixed datasets without interacting with the environment during training.

See all 290 papers

From Suboptimal to Optimal: One-Step Latent Flow Policies for Offline Reinforcement Learning
Oct 12, 2026Shenghua Lu, Zixuan Yang, Rongxu Zhao +3Offline RL Flow-Based Policy Learning
Jilin University
Diffusion Policy Improvement with Proposal-Conditioned Refinement Flows
Sep 29, 2026Junhyun Ha, Juho Lee, Byoungwoo ParkPolicy Optimization Offline RL
KAIST
Flow Actor-Critic for Offline Reinforcement Learning
Feb 20, 2026Jongseong Chae, Jongeui Park, Yongjae Shin +3Flow-Based Policy Learning Conservative RL
KAIST · UNIST

Oct 12, 2026J/K move · Enter open · S save

From Suboptimal to Optimal: One-Step Latent Flow Policies for Offline Reinforcement Learning

Shenghua Lu, Zixuan Yang, Rongxu Zhao, Bohao Qu+2

Jilin University

Offline RL Flow-Based Policy Learning Latent Action Learning

Open paper AskPDF arXiv

Offline reinforcement learning aims to learn effective policies from fixed datasets without environment interaction. Although generative models such as diffusion and flow matching enhance policy expressiveness, they typically start from uninformative Gaussian noise and generate actions through multi-step iteration, leading to high computational costs and failing to effectively leverage suboptimal trajectories in datasets. We propose MeanFlow2Better (MF2B), which reformulates offline RL as generative decision chunk optimization from current performance status to better performance status in a high-level latent decision space. MF2B maps trajectory chunks to high-level latent decision representations through an encoder, placing generation starting points within a structured decision space sculpted by behavioral data. The model learns a shared conditional mean velocity field in this space, constructing two flow mappings: from Gaussian-Induced Decision Sources toward positive behavioral decision regions, and from suboptimal decisions toward high-value decision regions. Both flow mappings complete decision refinement through one-step ODE integration over the unit time interval

Δt=1

, requiring neither multi-step iterative sampling nor backpropagation through time. Furthermore, we propose hierarchical chunk-level advantage to guide the flow direction toward high-return regions, where the high-level decisions cover multiple low-level actions, leading to sequence length compression from

T

K \approx T/H

and significantly lower computational overhead. On the D4RL Gym-MuJoCo benchmark, MF2B achieves an average score of 90.91, matching or surpassing strong baselines.

Abstract

Offline RL

Explore similar work

From Suboptimal to Optimal: One-Step Latent Flow Policies for Offline Reinforcement Learning

Diffusion Policy Improvement with Proposal-Conditioned Refinement Flows

Flow Actor-Critic for Offline Reinforcement Learning