RoMAN-Flow: Taming Autoregressive Normalizing Flows for Offline Reinforcement Learning in Robotic Manipulation

Shaoxuan Wang; Guangting Zheng; Rui Huang; Zhipeng Tang; Sha Zhang; Jiajun Deng; Yanyong Zhang

cs.CVAug 20, 2026

RoMAN-Flow: Taming Autoregressive Normalizing Flows for Offline Reinforcement Learning in Robotic Manipulation

Authors: Shaoxuan Wang, Guangting Zheng, Rui Huang, Zhipeng Tang, Sha Zhang, Jiajun Deng, Yanyong Zhang

Organizations: University of Science and Technology of China · The Chinese University of Hong Kong

Abstract

Offline reinforcement learning improves robotic policies using previously collected data without further environment interaction. Yet prevalent diffusion- and flow-matching robot policies lack tractable likelihoods, limiting their use in likelihood-based offline RL post-training. AR-NFs offer both expressive action modeling and exact likelihood evaluation, but their sequential sampling incurs substantial sampling overhead during policy optimization and deployment. We present RoMAN-Flow (Robotic Manipulation with Autoregressive Normalizing Flows), an offline reinforcement learning framework that makes AR-NF policies practical for robotic manipulation by addressing this sampling bottleneck in both stages. During policy optimization, RoMAN-Flow employs a sampling-free, advantage-weighted likelihood objective that assigns higher likelihood to high-advantage actions from the offline dataset without sampling from the autoregressive policy. For efficient deployment, it distills the optimized autoregressive policy into a one-step action generator, enabling low-latency action prediction. Experiments across multiple simulated manipulation benchmarks and real-world robotic platforms demonstrate that RoMAN-Flow achieves competitive policy performance while substantially reducing inference latency. Code is available at https://github.com/konnyaku28/RoMAN-Flow.

Figures & tables

Figure 1: Overview of RoMAN-Flow. RoMAN-Flow adopts an invertible AR-NF architecture that maps continuous action chunks to latent variables conditioned on multimodal inputs. For imitation learning, it maximizes the exact conditional likelihood of demonstrated action chunks under a Gaussian latent prior. For action generation, it samples latent variables from the Gaussian prior and recovers action chunks through the autoregressive inverse transformation. For likelihood-based reinforcement learning, NF-IQL directly increases the exact likelihoods of high-advantage offline action chunks without autoregressive policy sampling.

Figure 2: One-step policy distillation. The student model aligns with the teacher’s intermediate reverse states and reconstructs the final action chunk in a single forward pass.

Figure 3: Qualitative results of RoMAN-Flow on MetaWorld, LIBERO-Long, RoboMimic, and real-world tasks. Each row shows selected frames from a rollout, while the bottom-right inset presents our real-robot platform with a seven-DoF Franka arm and a twelve-DoF XHand dexterous hand.

Method	#Params	Easy	Medium	Hard	Very Hard	Average
Diffusion Policy ( Chi et al. 2025 )	157M	23.1	10.7	1.9	6.1	10.5
TinyVLA ( Wen et al. 2025 )	1.3B	77.6	21.5	11.4	15.8	31.6
SmolVLA ( Shukor et al. 2025 )	2.25B	87.1	51.8	70.0	64.0	68.2
$\pi_{0}$ ( Black et al. 2024 )	3.3B	77.9	51.8	53.3	20.0	50.8
$\pi_{0}$ + Flow-SDE ( Chen et al. 2025 )	3.3B	92.1	74.6	61.7	84.0	78.1
$\pi_{0.5}$ ( Physical Intelligence et al. 2025 )	2.3B	68.2	37.3	41.7	28.0	43.8

Table 1: Success rates (%) on MetaWorld-MT50. Avg. is the unweighted mean over the four difficulty groups. Bold and underlined numerical entries denote the best and second-best results, respectively.

Method	#Params	Spatial	Object	Goal	Long	Average
Diffusion Policy ( Chi et al. 2025 )	157M	78.3	92.5	68.3	50.5	72.4
OpenVLA ( Kim et al. 2024 )	7.5B	84.7	88.4	79.2	53.7	76.5
NORA-Long ( Hung et al. 2025 )	3B	92.2	95.4	89.4	74.6	87.9
SmolVLA ( Shukor et al. 2025 )	2.25B	93.0	94.0	91.0	77.0	88.8
GR00T-N1 ( Bjorck et al. 2025 )	2.2B	94.4	97.6	93.0	90.6	93.9
$\pi_{0}$ ( Black et al. 2024 )	3.3B	96.8	98.8	95.8	85.2	94.2

Table 2: Success rates (%) on the four LIBERO task suites. Average is the unweighted mean across suites. Bold and underlined numerical entries denote the best and second-best results, respectively.

Method	Pick_beaker	Pick_cylinder	Place_beaker	Balance	Average
Diffusion Policy ( Chi et al. 2025 )	6	0	0	0	1.5
$\pi_{0}$ ( Black et al. 2024 )	80	3	93	73	62.3
$\pi_{0.5}$ ( Physical Intelligence et al. 2025 )	93	10	97	87	71.8
RoMAN-Flow (IL)	36	20	100	73	57.3
RoMAN-Flow (NF-IQL)	100	33	100	93	81.5
RoMAN-Flow (One-Step)	78	20	80	100	69.5

Table 3: Success rates (%) on four real-robot manipulation tasks. Average denotes the unweighted mean across the four tasks.

Method	Lift	Can	Square
SERNF (IL)	$79\pm 0$	$96\pm 0$	$61\pm 4$
SERNF (TD3+BC)	$91\pm 4$	$96\pm 2$	$68\pm 5$
RoMAN-Flow (IL)	$\mathbf{100\pm 0}$	$\mathbf{97\pm 2}$	$80\pm 3$
RoMAN-Flow (NF-IQL)	$\mathbf{100\pm 0}$	$96\pm 3$	$\mathbf{85\pm 3}$
RoMAN-Flow (One-Step)	$\mathbf{100\pm 0}$	$95\pm 2$	$80\pm 3$

Table 4: Success rates (%) on RoboMimic MH. Results are reported as the mean and standard deviation over four training seeds, with 100 evaluation rollouts per seed.

Figure 4: Speed–performance trade-off on LIBERO-Long with action horizon

H=16

. Each point reports the average success rate and action-chunk generation latency over the ten LIBERO-Long tasks, with marker area proportional to the number of inference-time policy parameters.

Size	Dim.	Heads	Layer Depth	#Params	SR
S	384	6	$[2,2,2,2,2,2]$	33.8M	78.0
B	768	12	$[2,2,2,2,2,2]$	134.8M	77.0
L	1024	16	$[2,2,2,2,2,14]$	466.2M	78.0
XL	1152	16	$[2,2,2,2,2,18]$	685.5M	85.0

Table 5: The capacity ablation of AR-NF actor on RoboMimic Square-MH. SR denotes success rate (%).

Model configuration
	RoMAN-Flow (IL / NF-IQL)				RoMAN-Flow (One-Step)
Setting	LIBERO	MT50	RoboMimic	Real Robot	Default
Hidden dimension	1152	1152	1152	1152	512
Attention heads	16	16	16	16	8
Flow / ViT blocks	6	6	6	6	7
Inference parameters	1.45B	1.45B	0.95B	1.45B	0.56B / 78.2M

Table 6: Model configurations and training hyperparameters. Paired entries under RoMAN-Flow (IL / NF-IQL) report the IL and NF-IQL settings, respectively, while single entries are shared by both stages. The One-Step column reports the default distillation configuration, with its two parameter counts corresponding to the SmolVLM- and ResNet-based students. A dash denotes an inapplicable setting.

Figure 5: Qualitative results of RoMAN-Flow on representative MetaWorld-MT50 manipulation tasks.

Figure 6: Qualitative rollouts of RoMAN-Flow on representative tasks from the LIBERO benchmark.

Figure 7: Qualitative rollouts of RoMAN-Flow on the Lift, Can, and Square tasks from RoboMimic MH.

Robotic RL

Robotic reinforcement learning uses trial-and-error learning to help robots control, navigate, and manipulate objects.

See all 299 papers

May 3, 2026

Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning

We propose Flow-Anchored Noise-conditioned Q-Learning (FAN), a highly efficient and high-performing offline reinforcement learning (RL) algorithm. Recent work has shown that expressive flow policies and distributional critics improve offline RL performance, but at a high computational cost. Specifically, flow policies require iterative sampling to produce a single action, and distributional critics require computation over multiple samples (e.g., quantiles) to estimate value. To address these inefficiencies while maintaining high performance, we introduce FAN. Our method employs a behavior regularization technique that uses a single flow policy iteration and requires a single Gaussian noise sample for distributional critics. Our theoretical analysis of convergence and performance bounds demonstrates that these simplifications not only improve efficiency but also lead to superior task performance. Experiments on robotic manipulation and locomotion tasks demonstrate that FAN achieves state-of-the-art performance while significantly reducing both training and inference runtimes. We release our code at https://github.com/brianlsy98/FAN.

Sungyoung Lee, Dohyeong Kim, Eshan Balachandar +2

The University of Texas at Austin, Austin, TX, USA · Independent Researcher, Seoul, South Korea.

Distributional RL Q-Learning Robotic RL Offline RL Policy Learning

Ask PDF

May 29, 2025

Normalizing Flows are Capable Models for Continuous Control

Modern reinforcement learning (RL) algorithms have found success by using powerful probabilistic models, such as transformers, energy-based models, and diffusion/flow-based models. To this end, RL researchers often choose to pay the price of accommodating these models into their algorithms -- diffusion models are expressive, but are computationally intensive due to their reliance on solving differential equations, while autoregressive transformer models are scalable but typically require learning discrete representations. Normalizing flows (NFs), by contrast, seem to provide an appealing alternative, as they enable likelihoods and sampling without solving differential equations or autoregressive architectures. However, their potential in RL has received limited attention, partly due to the prevailing belief that normalizing flows lack sufficient expressivity. We show that this is not the case. Building on recent work in NFs, we propose a single NF architecture which integrates seamlessly into RL algorithms, serving as a policy, Q-function, and occupancy measure. Our approach leads to much simpler algorithms, and achieves higher performance in imitation learning, offline, goal conditioned RL and unsupervised RL.

Raj Ghugare, Benjamin Eysenbach

Department of Computer Science Princeton University

Normalizing Flows Reinforcement Learning Unsupervised RL Continuous Control Flow-Based Generative Modeling Flow-Based Policy Learning

Ask PDF

Mar 12, 2026

NFPO: Stabilized Policy Optimization of Normalizing Flow for Robotic Policy Learning

Deep Reinforcement Learning (DRL) has experienced significant advancements in recent years and has been widely used in many fields. In DRL-based robotic policy learning, however, current de facto policy parameterization is still multivariate Gaussian (with diagonal covariance matrix), which lacks the ability to model multi-modal distribution. In this work, we explore the adoption of a modern network architecture, i.e. Normalizing Flow (NF) as the policy parameterization for its ability of multi-modal modeling, closed form of log probability and low computation and memory overhead. However, naively training NF in online Reinforcement Learning (RL) usually leads to training instability. We provide a detailed analysis for this phenomenon and successfully address it via simple but effective technique. With extensive experiments in multiple simulation environments, we show our method, NFPO could obtain robust and strong performance in widely used robotic learning tasks and successfully transfer into real-world robots.

Diyuan Shi, Yiqi Tang, Zifeng Zhuang +1

Zhejiang University · Westlake University

RL for Robotics Robotic RL Normalizing Flows

Ask PDF

Abstract

Robotic RL

Explore similar work

Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning

Normalizing Flows are Capable Models for Continuous Control

NFPO: Stabilized Policy Optimization of Normalizing Flow for Robotic Policy Learning