Flow Actor-Critic for Offline Reinforcement Learning

Jongseong Chae; Jongeui Park; Yongjae Shin; Gyeongmin Kim; Seungyul Han; Youngchul Sung

cs.LGFeb 20, 2026

Flow Actor-Critic for Offline Reinforcement Learning

Authors: Jongseong Chae, Jongeui Park, Yongjae Shin, Gyeongmin Kim, Seungyul Han, Youngchul Sung

Organizations: KAIST · UNIST

Abstract

The dataset distributions in offline reinforcement learning (RL) often exhibit complex and multi-modal distributions, necessitating expressive policies to capture such distributions beyond widely-used Gaussian policies. To handle such complex and multi-modal datasets, in this paper, we propose Flow Actor-Critic, a new actor-critic method for offline RL, based on recent flow policies. The proposed method not only uses the flow model for actor as in previous flow policies but also exploits the expressive flow model for conservative critic acquisition to prevent Q-value explosion in out-of-data regions. To this end, we propose a new form of critic regularizer based on the flow behavior proxy model obtained as a byproduct of flow-based actor design. Leveraging the flow model in this joint way, we achieve new state-of-the-art performance for test datasets of offline RL including the D4RL and recent OGBench benchmarks.

Figures & tables

Figure 1: BC models on a synthetic four-component Gaussian mixture dataset: Top row - samples from each BC model. Bottom row - log-density or ELBO plot.

Figure 2: weight

w^{\hat{\beta}}

	Gaussian Policies			Diffusion Policies			Flow Policies
Task Category	BC	IQL	ReBRAC	IDQL	SRPO	CAC	FAWAC	FBRAC	IFQL	FQL	FAC (Ours)
antmaze-large-navigate	10.6	53.4	80.8	20.8	10.6	32.8	6.4	60.2	28.0	78.6	92.6 $\pm$ 2.5
antmaze-giant-navigate	0.2	4.0	26.2	0.0	0.0	0.0	0.0	3.8	2.6	8.6	23.0 $\pm$ 5.1
humanoidmaze-medium-navigate	2.0	32.8	21.8	0.8	1.4	52.8	19.4	38.4	60.4	57.4	75.6 $\pm$ 3.6
humanoidmaze-large-navigate	0.4	2.4	2.6	0.6	0.2	0.6	0.2	2.2	11.0	4.2	8.3 $\pm$ 5.5
antsoccer-arena-navigate	1.0	8.4	0.0	11.8	1.0	1.8	12.4	16.0	33.2	60.2	67.7 $\pm$ 2.9

Table 1: Evaluation over 55 singletasks of the OGBench. For each task category, we report the final performance averaged across its 5 singletasks, over 8 seeds (4 seeds for pixel-based tasks), with

\pm

indicating the standard deviation. Full evaluation results on the 55 singletasks are in Table 3 .

	Gaussian Policies							Diffusion Policies			Flow Policies
MuJoCo Tasks	TD3+BC	IQL	CQL	MCQ	EPQ	SPOT	ReBRAC	IDQL	SRPO	CAC	QIPO-OT	FQL	FAC (Ours)
halfcheetah-m	48.3	47.4	44.0	64.3	67.3	58.4	65.6	51.0	60.4	69.1	54.2	60.3 $\pm$ 1.1	65.0 $\pm$ 1.5
hopper-m	59.3	66.3	58.5	78.4	101.3	86.0	102.0	65.4	95.5	80.7	94.1	68.1 $\pm$ 3.4	91.9 $\pm$ 3.9
walker2d-m	83.7	78.3	72.5	91.0	87.8	86.4	82.5	82.5	84.4	83.1	87.6	77.2 $\pm$ 2.5	85.2 $\pm$ 0.9
halfcheetah-mr	44.6	44.2	45.5	56.8	62.0	52.2	51.0	45.9	51.4	58.7	48.0	49.3 $\pm$ 0.5	55.4 $\pm$ 2.7
hopper-mr	60.9	94.7	95.0	101.6	97.8	100.2	98.1	92.1	101.2	99.7	101.3	49.8 $\pm$ 7.2	99.1 $\pm$ 0.9

Table 2: Evaluation on 23 tasks of the D4RL. We report the final performance averaged over 8 seeds, with

\pm

indicating the standard deviation. For MuJoCo datasets, we use the following abbreviations: m for medium , mr for medium-replay , me for medium-expert .

Panel of Figure 3 — Figure 3: Empirical analysis of components. (a) Effect of flow-based critic penalization (CP) on performance under $N$ candidate actions. (b) Performances across actor regularization coefficient $\lambda$ and critic penalization coefficient $\alpha$ . (c) Influence by the fidelity of flow behavior proxy under flow steps $T$ . (d) Performance of weight function designs. (e) Performance under different $\epsilon$ schemes.

Figure 4: Offline-to-online results, averaged over 8 seeds with standard deviation. The gray shaded region indicates the offline phase, and the online fine-tuning phase is left unshaded. Full plots on 15 tasks are in Fig. 7 .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Flow behavior proxy model with different step counts of the Euler method.

Panel of Figure 6 — Figure 6: A comparison of FAC and baselines in a continuous action bandit problem. (a) Dataset and true Q value of the problem. (b) Comparison of baselines and FAC. (c) The IS ratio in the critic penalization of SVR. (d) Results of FQL under two actor regularization coefficients $\lambda=\{1,10\}$ .

	Gaussian Policies			Diffusion Policies			Flow Policies
Task	BC	IQL	ReBRAC	IDQL	SRPO	CAC	FAWAC	FBRAC	IFQL	FQL	FAC (Ours)
antmaze-large-navigate-singletask-task1-v0 ( $\ast$ )	0	48	91	0	0	42	1	70	24	80	94.0 $\pm$ 3.0
antmaze-large-navigate-singletask-task2-v0	6	42	88	14	4	1	0	35	8	57	86.0 $\pm$ 5.7
antmaze-large-navigate-singletask-task3-v0	29	72	51	26	3	49	12	83	52	93	97.5 $\pm$ 2.1
antmaze-large-navigate-singletask-task4-v0	8	51	84	62	45	17	10	37	18	80	89.5 $\pm$ 7.7
antmaze-large-navigate-singletask-task5-v0	10	54	90	2	1	55	9	76	38	83	96.0 $\pm$ 4.8

Appendix

Table 3: Full offline RL evaluation results on the OGBench. (

\ast

) indicates the default task in each task category. We report the final performance averaged over 8 seeds (4 seeds for pixel-based tasks)

	Transformer based				SSM based	MLP based
MuJoCo Tasks	DT	DC	QDT	Reinformer	DM	FAC (Ours)
halfcheetah-m	42.6	43.0	42.3	42.9	43.8	65.0 $\pm$ 1.5
hopper-m	67.6	92.5	66.5	81.6	98.5	91.9 $\pm$ 3.9
walker2d-m	74.0	79.2	67.1	80.5	80.3	85.2 $\pm$ 0.9
halfcheetah-mr	36.6	41.3	35.6	39.0	40.8	55.4 $\pm$ 2.7
hopper-mr	82.7	94.2	52.1	83.3	89.1	99.1 $\pm$ 0.9

Appendix

Table 4: Comparison with sequence model based baselines. For MuJoCo datasets, we use the following abbreviations: m for medium , mr for medium-replay , me for medium-expert .

Figure 7: Offline-to-online results on 15 tasks, averaged over 8 seeds with standard deviation. The gray shaded region indicates the offline phase, and the online fine-tuning phase is left unshaded.

Hyperparameters	Value
Learning rate	0.0003
Optimizer	Adam ( Kingma & Ba, 2014 )
Gradient Steps	1000000
Minibatch size	256
MLP dimension	[512, 512, 512, 512]
Nonlinearity	GELU ( Hendrycks & Gimpel, 2016 )

Appendix

Table 5: Hyperparameters for FQL on the D4RL-MuJoCo domain.

Tasks (Datasets)	BC coefficient $\lambda$
halfcheetah-medium-v2	3.0
hopper-medium-v2	100.0
walker2d-medium-v2	300.0
halfcheetah-medium-replay-v2	30.0
hopper-medium-replay-v2	100.0
walker2d-medium-replay-v2	300.0

Appendix

Table 6: BC Coefficient

\lambda

for FQL on the D4RL-MuJoCo domain.

Hyperparameters	Value
Learning rate	0.0003
Optimizer	Adam ( Kingma & Ba, 2014 )
Gradient Steps	1M (default), 500K ( OGBench :pixel-based tasks)
Minibatch size	256
Epochs for flow proxy model	250 (default),
	125 ( D4RL:MuJoCo, OGBench:pixel-based tasks )

Appendix

Table 7: Hyperparameters for our method

Table 8: Critic penalization coefficient

\alpha

and actor regularization coefficient

\lambda

of FAC

Figure 8: Computational costs of FAC and baselines

Explore similar work

Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning
May 3, 2026Sungyoung Lee, Dohyeong Kim, Eshan Balachandar +2Distributional RL Q-Learning
The University of Texas at Austin, Austin, TX, USA · Independent Researcher, Seoul, South Korea.
Simple Actors and Deep Critics for Scalable Reinforcement Learning
Aug 27, 2026Guhyeon Kang, Jaehwi Lee, Minhae KwonOffline RL Actor-Critic Methods
Sungkyunkwan University Republic of Korea · Department of Electrical and Computer Engineering, Sungkyunkwan University, Republic of Korea · Soongsil University Republic of Korea
Learning Multimodal One-step Flow Policy via Value-weighted Optimal Transport
Sep 14, 2026Jaehun Shon, Jinha Choi, Jongwook Jeon +1Offline RL
Department of Artificial Intelligence, Yonsei University

May 3, 2026J/K move · Enter open · S save

Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning

Sungyoung Lee, Dohyeong Kim, Eshan Balachandar, Zelal Su Mustafaoglu+1

The University of Texas at Austin, Austin, TX, USA · Independent Researcher, Seoul, South Korea.

Distributional RL Q-Learning Robotic RL Offline RL Policy Learning

Open paper AskPDF arXiv

We propose Flow-Anchored Noise-conditioned Q-Learning (FAN), a highly efficient and high-performing offline reinforcement learning (RL) algorithm. Recent work has shown that expressive flow policies and distributional critics improve offline RL performance, but at a high computational cost. Specifically, flow policies require iterative sampling to produce a single action, and distributional critics require computation over multiple samples (e.g., quantiles) to estimate value. To address these inefficiencies while maintaining high performance, we introduce FAN. Our method employs a behavior regularization technique that uses a single flow policy iteration and requires a single Gaussian noise sample for distributional critics. Our theoretical analysis of convergence and performance bounds demonstrates that these simplifications not only improve efficiency but also lead to superior task performance. Experiments on robotic manipulation and locomotion tasks demonstrate that FAN achieves state-of-the-art performance while significantly reducing both training and inference runtimes. We release our code at https://github.com/brianlsy98/FAN.

Abstract

Explore similar work

Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning

Simple Actors and Deep Critics for Scalable Reinforcement Learning

Learning Multimodal One-step Flow Policy via Value-weighted Optimal Transport