RoMAN-Flow: Taming Autoregressive Normalizing Flows for Offline Reinforcement Learning in Robotic Manipulation

Shaoxuan Wang; Guangting Zheng; Rui Huang; Zhipeng Tang; Sha Zhang; Jiajun Deng; Yanyong Zhang

cs.CVAug 20, 2026

RoMAN-Flow: Taming Autoregressive Normalizing Flows for Offline Reinforcement Learning in Robotic Manipulation

Authors: Shaoxuan Wang, Guangting Zheng, Rui Huang, Zhipeng Tang, Sha Zhang, Jiajun Deng, Yanyong Zhang

Organizations: University of Science and Technology of China · The Chinese University of Hong Kong

Abstract

Offline reinforcement learning improves robotic policies using previously collected data without further environment interaction. Yet prevalent diffusion- and flow-matching robot policies lack tractable likelihoods, limiting their use in likelihood-based offline RL post-training. AR-NFs offer both expressive action modeling and exact likelihood evaluation, but their sequential sampling incurs substantial sampling overhead during policy optimization and deployment. We present RoMAN-Flow (Robotic Manipulation with Autoregressive Normalizing Flows), an offline reinforcement learning framework that makes AR-NF policies practical for robotic manipulation by addressing this sampling bottleneck in both stages. During policy optimization, RoMAN-Flow employs a sampling-free, advantage-weighted likelihood objective that assigns higher likelihood to high-advantage actions from the offline dataset without sampling from the autoregressive policy. For efficient deployment, it distills the optimized autoregressive policy into a one-step action generator, enabling low-latency action prediction. Experiments across multiple simulated manipulation benchmarks and real-world robotic platforms demonstrate that RoMAN-Flow achieves competitive policy performance while substantially reducing inference latency. Code is available at https://github.com/konnyaku28/RoMAN-Flow.

Figures & tables

Figure 1: Overview of RoMAN-Flow. RoMAN-Flow adopts an invertible AR-NF architecture that maps continuous action chunks to latent variables conditioned on multimodal inputs. For imitation learning, it maximizes the exact conditional likelihood of demonstrated action chunks under a Gaussian latent prior. For action generation, it samples latent variables from the Gaussian prior and recovers action chunks through the autoregressive inverse transformation. For likelihood-based reinforcement learning, NF-IQL directly increases the exact likelihoods of high-advantage offline action chunks without autoregressive policy sampling.

Figure 2: One-step policy distillation. The student model aligns with the teacher’s intermediate reverse states and reconstructs the final action chunk in a single forward pass.

Figure 3: Qualitative results of RoMAN-Flow on MetaWorld, LIBERO-Long, RoboMimic, and real-world tasks. Each row shows selected frames from a rollout, while the bottom-right inset presents our real-robot platform with a seven-DoF Franka arm and a twelve-DoF XHand dexterous hand.

Method	#Params	Easy	Medium	Hard	Very Hard	Average
Diffusion Policy ( Chi et al. 2025 )	157M	23.1	10.7	1.9	6.1	10.5
TinyVLA ( Wen et al. 2025 )	1.3B	77.6	21.5	11.4	15.8	31.6
SmolVLA ( Shukor et al. 2025 )	2.25B	87.1	51.8	70.0	64.0	68.2
$\pi_{0}$ ( Black et al. 2024 )	3.3B	77.9	51.8	53.3	20.0	50.8
$\pi_{0}$ + Flow-SDE ( Chen et al. 2025 )	3.3B	92.1	74.6	61.7	84.0	78.1
$\pi_{0.5}$ ( Physical Intelligence et al. 2025 )	2.3B	68.2	37.3	41.7	28.0	43.8

Table 1: Success rates (%) on MetaWorld-MT50. Avg. is the unweighted mean over the four difficulty groups. Bold and underlined numerical entries denote the best and second-best results, respectively.

Method	#Params	Spatial	Object	Goal	Long	Average
Diffusion Policy ( Chi et al. 2025 )	157M	78.3	92.5	68.3	50.5	72.4
OpenVLA ( Kim et al. 2024 )	7.5B	84.7	88.4	79.2	53.7	76.5
NORA-Long ( Hung et al. 2025 )	3B	92.2	95.4	89.4	74.6	87.9
SmolVLA ( Shukor et al. 2025 )	2.25B	93.0	94.0	91.0	77.0	88.8
GR00T-N1 ( Bjorck et al. 2025 )	2.2B	94.4	97.6	93.0	90.6	93.9
$\pi_{0}$ ( Black et al. 2024 )	3.3B	96.8	98.8	95.8	85.2	94.2

Table 2: Success rates (%) on the four LIBERO task suites. Average is the unweighted mean across suites. Bold and underlined numerical entries denote the best and second-best results, respectively.

Method	Pick_beaker	Pick_cylinder	Place_beaker	Balance	Average
Diffusion Policy ( Chi et al. 2025 )	6	0	0	0	1.5
$\pi_{0}$ ( Black et al. 2024 )	80	3	93	73	62.3
$\pi_{0.5}$ ( Physical Intelligence et al. 2025 )	93	10	97	87	71.8
RoMAN-Flow (IL)	36	20	100	73	57.3
RoMAN-Flow (NF-IQL)	100	33	100	93	81.5
RoMAN-Flow (One-Step)	78	20	80	100	69.5

Table 3: Success rates (%) on four real-robot manipulation tasks. Average denotes the unweighted mean across the four tasks.

Method	Lift	Can	Square
SERNF (IL)	$79\pm 0$	$96\pm 0$	$61\pm 4$
SERNF (TD3+BC)	$91\pm 4$	$96\pm 2$	$68\pm 5$
RoMAN-Flow (IL)	$\mathbf{100\pm 0}$	$\mathbf{97\pm 2}$	$80\pm 3$
RoMAN-Flow (NF-IQL)	$\mathbf{100\pm 0}$	$96\pm 3$	$\mathbf{85\pm 3}$
RoMAN-Flow (One-Step)	$\mathbf{100\pm 0}$	$95\pm 2$	$80\pm 3$

Table 4: Success rates (%) on RoboMimic MH. Results are reported as the mean and standard deviation over four training seeds, with 100 evaluation rollouts per seed.

Figure 4: Speed–performance trade-off on LIBERO-Long with action horizon

H=16

. Each point reports the average success rate and action-chunk generation latency over the ten LIBERO-Long tasks, with marker area proportional to the number of inference-time policy parameters.

Size	Dim.	Heads	Layer Depth	#Params	SR
S	384	6	$[2,2,2,2,2,2]$	33.8M	78.0
B	768	12	$[2,2,2,2,2,2]$	134.8M	77.0
L	1024	16	$[2,2,2,2,2,14]$	466.2M	78.0
XL	1152	16	$[2,2,2,2,2,18]$	685.5M	85.0

Table 5: The capacity ablation of AR-NF actor on RoboMimic Square-MH. SR denotes success rate (%).

Model configuration
	RoMAN-Flow (IL / NF-IQL)				RoMAN-Flow (One-Step)
Setting	LIBERO	MT50	RoboMimic	Real Robot	Default
Hidden dimension	1152	1152	1152	1152	512
Attention heads	16	16	16	16	8
Flow / ViT blocks	6	6	6	6	7
Inference parameters	1.45B	1.45B	0.95B	1.45B	0.56B / 78.2M

Table 6: Model configurations and training hyperparameters. Paired entries under RoMAN-Flow (IL / NF-IQL) report the IL and NF-IQL settings, respectively, while single entries are shared by both stages. The One-Step column reports the default distillation configuration, with its two parameter counts corresponding to the SmolVLM- and ResNet-based students. A dash denotes an inapplicable setting.

Figure 5: Qualitative results of RoMAN-Flow on representative MetaWorld-MT50 manipulation tasks.

Figure 6: Qualitative rollouts of RoMAN-Flow on representative tasks from the LIBERO benchmark.

Figure 7: Qualitative rollouts of RoMAN-Flow on the Lift, Can, and Square tasks from RoboMimic MH.

Robotic RL

Robotic reinforcement learning uses trial-and-error learning to help robots control, navigate, and manipulate objects.

See all 299 papers

Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning
May 3, 2026Sungyoung Lee, Dohyeong Kim, Eshan Balachandar +2Distributional RL Q-Learning
The University of Texas at Austin, Austin, TX, USA · Independent Researcher, Seoul, South Korea.
Normalizing Flows are Capable Models for Continuous Control
May 29, 2025Raj Ghugare, Benjamin EysenbachNormalizing Flows Reinforcement Learning
Department of Computer Science Princeton University
NFPO: Stabilized Policy Optimization of Normalizing Flow for Robotic Policy Learning
Mar 12, 2026Diyuan Shi, Yiqi Tang, Zifeng Zhuang +1RL for Robotics Robotic RL
Zhejiang University · Westlake University

May 3, 2026J/K move · Enter open · S save

Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning

Sungyoung Lee, Dohyeong Kim, Eshan Balachandar, Zelal Su Mustafaoglu+1

The University of Texas at Austin, Austin, TX, USA · Independent Researcher, Seoul, South Korea.

Distributional RL Q-Learning Robotic RL Offline RL Policy Learning

Open paper AskPDF arXiv

We propose Flow-Anchored Noise-conditioned Q-Learning (FAN), a highly efficient and high-performing offline reinforcement learning (RL) algorithm. Recent work has shown that expressive flow policies and distributional critics improve offline RL performance, but at a high computational cost. Specifically, flow policies require iterative sampling to produce a single action, and distributional critics require computation over multiple samples (e.g., quantiles) to estimate value. To address these inefficiencies while maintaining high performance, we introduce FAN. Our method employs a behavior regularization technique that uses a single flow policy iteration and requires a single Gaussian noise sample for distributional critics. Our theoretical analysis of convergence and performance bounds demonstrates that these simplifications not only improve efficiency but also lead to superior task performance. Experiments on robotic manipulation and locomotion tasks demonstrate that FAN achieves state-of-the-art performance while significantly reducing both training and inference runtimes. We release our code at https://github.com/brianlsy98/FAN.

Abstract

Robotic RL

Explore similar work

Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning

Normalizing Flows are Capable Models for Continuous Control

NFPO: Stabilized Policy Optimization of Normalizing Flow for Robotic Policy Learning