NFPO: Stabilized Policy Optimization of Normalizing Flow for Robotic Policy Learning

Diyuan Shi; Yiqi Tang; Zifeng Zhuang; Donglin Wang

cs.ROMar 12, 2026

NFPO: Stabilized Policy Optimization of Normalizing Flow for Robotic Policy Learning

Authors: Diyuan Shi, Yiqi Tang, Zifeng Zhuang, Donglin Wang

Organizations: Zhejiang University · Westlake University

Abstract

Deep Reinforcement Learning (DRL) has experienced significant advancements in recent years and has been widely used in many fields. In DRL-based robotic policy learning, however, current de facto policy parameterization is still multivariate Gaussian (with diagonal covariance matrix), which lacks the ability to model multi-modal distribution. In this work, we explore the adoption of a modern network architecture, i.e. Normalizing Flow (NF) as the policy parameterization for its ability of multi-modal modeling, closed form of log probability and low computation and memory overhead. However, naively training NF in online Reinforcement Learning (RL) usually leads to training instability. We provide a detailed analysis for this phenomenon and successfully address it via simple but effective technique. With extensive experiments in multiple simulation environments, we show our method, NFPO could obtain robust and strong performance in widely used robotic learning tasks and successfully transfer into real-world robots.

Figures & tables

1-1 Algorithm	Multi-modality	On-policy	Memory & Computation	Real World Deployment
FastTD3	✗	✗	✗	✓
Meow	✓	✗	✓	✗
MaxEntDP	✓	✗	✗	✗
GenPO	✓	✓	✗	✗
FPO α	✓	✓	✓	✗
Ours (NFPO)	✓	✓	✓	✓

Table 1: Comparison of NFPO to related methods. Detailed explanations of how we choose ✓or ✗ and the references are provided in Section A.1 .

Figure 2: Training NFs in Unitree Gym’s g1. The experiment is in 3 seeds with 95% confidence interval. s_none early stopped due to training instability, s_clip and s_tanh overlaps in determinant plot.

Figure 3: Studies in various factors. Experiments are in 3 seeds with 95% confidence interval.

Figure 4: Studies in various factors. Experiments are in 3 seeds with 95% confidence interval.

Figure 5: Learning curves of various methods on representative robotic tasks. The experiments are in 10 seeds. The errorbars are 95% confidence interval.

Figure 6: Difference in generated trajectory of NFPO and PPO in gridworld.

Panel of Figure 7 — Figure 7: Difference in generated trajectories of NFPO and PPO in UR10 Reaching.

Figure 8: Learning curves of Meow and NFPO in 2 environments of Unitree RL Gym. Legend is meow- alpha - polyak where alpha and polyak are 2 tuned hyperparameters.

1-1 Task	FPO	NFPO
AlohaHandOver	✓	✗
G1JoystickRoughTerrain	✗	✓
Go1Footstand	✗	✓
Go1Handstand	✓	✓
Go1JoystickRoughTerrain	✗	✗
PandaOpenCabinet	✗	✓

Table 2: NFPO and FPO ’s accomplished tasks in Mujoco playground.

Panel of Figure 9 — Figure 9: Snapshoots of real world deployment video.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: Performance of various online RL methods on Mujoco environments (5 seeds).

Panel of Figure 12 — Figure S2: Performance of NFPO over various number of layers and hidden dimensions (5 seeds).

Figure S3: Performance of Meow over various tuned polyak and alpha . The experiments are in 3 seeds.

1-1 \Block 1-2Hyperparameter		Value	Remarks
\Block 9-1Shared	GAE $\lambda$	0.95
	discount $\gamma$	0.99
	value loss coefficient	1.0
	grad clip	1.0
	learning epoch	5
	learning minibath	4

Appendix

Table S1: Hyperparameter settings

1-1 Algorithm	Runtime (s)	Runtime (h)	Ratio
PPO	13500.94 $\pm$ 628.29	3.75 $\pm$ 0.17	1
NFPO	16036.06 $\pm$ 795.74	4.45 $\pm$ 0.22	1.19

Appendix

Table S2: Runtime of PPO and NFPO on Unitree Gym’s g1. Calculated using 10 seeds.

RL for Robotics

Reinforcement learning for robotics trains robots to act through interaction, from locomotion and navigation to dexterous manipulation.

See all 252 papers

RoMAN-Flow: Taming Autoregressive Normalizing Flows for Offline Reinforcement Learning in Robotic Manipulation
Aug 20, 2026Shaoxuan Wang, Guangting Zheng, Rui Huang +4Robotic RL Offline RL
University of Science and Technology of China · The Chinese University of Hong Kong
Normalizing Flows are Capable Models for Continuous Control
May 29, 2025Raj Ghugare, Benjamin EysenbachNormalizing Flows Reinforcement Learning
Department of Computer Science Princeton University
Drift-Based Policy Optimization: Native One-Step Policy Learning for Online Robot Control
Apr 4, 2026Yuxuan Gao, Yedong Shen, Shiqi Zhang +6Diffusion Policy Robotic RL
University of Science and Technology of China, Hefei 230026, China. · LYNSENSE, Shanghai, China. · iFLYTEK, Hefei 230088, China.

Aug 20, 2026J/K move · Enter open · S save

RoMAN-Flow: Taming Autoregressive Normalizing Flows for Offline Reinforcement Learning in Robotic Manipulation

Shaoxuan Wang, Guangting Zheng, Rui Huang, Zhipeng Tang+3

University of Science and Technology of China · The Chinese University of Hong Kong

Robotic RL Offline RL Robotic Manipulation

Open paper AskPDF arXiv

Offline reinforcement learning improves robotic policies using previously collected data without further environment interaction. Yet prevalent diffusion- and flow-matching robot policies lack tractable likelihoods, limiting their use in likelihood-based offline RL post-training. AR-NFs offer both expressive action modeling and exact likelihood evaluation, but their sequential sampling incurs substantial sampling overhead during policy optimization and deployment. We present RoMAN-Flow (Robotic Manipulation with Autoregressive Normalizing Flows), an offline reinforcement learning framework that makes AR-NF policies practical for robotic manipulation by addressing this sampling bottleneck in both stages. During policy optimization, RoMAN-Flow employs a sampling-free, advantage-weighted likelihood objective that assigns higher likelihood to high-advantage actions from the offline dataset without sampling from the autoregressive policy. For efficient deployment, it distills the optimized autoregressive policy into a one-step action generator, enabling low-latency action prediction. Experiments across multiple simulated manipulation benchmarks and real-world robotic platforms demonstrate that RoMAN-Flow achieves competitive policy performance while substantially reducing inference latency. Code is available at https://github.com/konnyaku28/RoMAN-Flow.

Abstract

RL for Robotics

Explore similar work

RoMAN-Flow: Taming Autoregressive Normalizing Flows for Offline Reinforcement Learning in Robotic Manipulation

Normalizing Flows are Capable Models for Continuous Control

Drift-Based Policy Optimization: Native One-Step Policy Learning for Online Robot Control