Generative Flow Networks for continuous scenarios (CFlowNets) have shown promise in solving sequential decision-making tasks by learning stochastic policies using a flow and a retrieval network. Despite their demonstrated efficiency compared to state-of-the-art Reinforcement Learning (RL) algorithms, their practical application in robotic control tasks is constrained by the reliance on pre-training the retrieval network. This dependency poses challenges in dynamic robotic environments, where pre-training data may not be readily available or representative of the current environment. This paper introduces WINFlowNets, a novel CFlowNets framework that enables the co-training of flow and retrieval networks. WINFlowNets begins with a warm-up phase for the retrieval network to bootstrap its policy, followed by a shared training architecture and a shared replay buffer for co-training both networks. Experiments in simulated robotic environments demonstrate that WINFlowNets surpasses CFlowNets and state-of-the-art RL algorithms in terms of average reward and training stability. Furthermore, WINFlowNets exhibits strong adaptive capability in fault environments, making it suitable for tasks that demand quick adaptation with limited sample data. These findings highlight WINFlowNets' potential for deployment in dynamic and malfunction-prone robotic systems, where traditional pre-training or sample inefficient data collection may be impractical.
Figures & tables
Figure 1: Main concepts of WINFlowNets.
Figure 2: The architecture of WINFlowNets.
Figure 3: Visualization of the Reacher-v2 robotic arm.
Figure 4: Average reward of CFlowNets, WINFlowNets, and RL algorithms in the normal Reacher-v2 environment.
Model
Final Performance
Sample Efficiency
SAC
−7.89±0.16
0.67
PPO
−9.50±0.37
3.39
DDPG
−9.55±0.44
5.20
CFlowNets
−3.70±0.05
0.10
WINFlowNets
−2.39±0.17
0.72
Table 1: Average final performance ± standard deviation and sample efficiency (in millions of timesteps) of WINFlowNets and baseline models in the normal Reacher-v2 environment. Bold entries indicate the highest-performing model.
Figure 5: Average reward of CFlowNets, WINFlowNets, and RL algorithms in the Reacher-v2 environment under faulty conditions.
Model
Faulty Environment
AD
Reduced ROM
Final Perf.
Sample Eff.
Final Perf.
Sample Eff.
SAC
−9.69±0.19
0.11
−6.16±0.03
0.37
CFlowNets
−9.01±0.17
0.32
−8.50±0.08
0.11
WINFlowNets
−8.25±0.19
0.24
−5.25±0.12
0.12
Table 2: Average final performance ± standard deviation and sample efficiency (in millions of timesteps) of WINFlowNets, CFlowNets, and SAC in two faulty environments. Bold entries indicate the best-performing model in each setting.
Figure 6: Average reward for Vanilla CFlowNets and WINFlowNets variants on our normal environment.
Figure 7: Confidence interval width of WINFlowNets average reward over time in three environments.
Flow-matching policies have emerged as an effective policy parameterization for robot learning. They iteratively generate actions from noise, enabling highly expressive modeling of complex and multimodal action distributions. However, prior works observed that scaling these policies with value-gradient reinforcement learning (RL) often leads to training instability. Existing methods attribute this instability to iterative generation and therefore avoid end-to-end value-gradient optimization by sacrificing iterative generation, high expressiveness, or value-gradient optimization. Contrary to prior belief, we show the instability does not stem from iterative generation itself, but from the vanilla sampling strategy originally designed for behavior cloning, which becomes brittle under value-gradient RL. Motivated by this insight, we propose VINE, an RL-oriented sampling method that enables stable end-to-end value-gradient optimization for flow-matching policies. Instead of following a single flow trajectory, VINE reconstructs a new interpolation state at every denoising step, creating a stable differentiable path for value-gradient propagation while remaining compatible with the original flow-matching denoising process. As a result, VINE preserves the expressiveness and iterative generation of flow-matching without sacrificing end-to-end value-gradient optimization. Despite performing end-to-end backpropagation through all ten denoising steps, VINE achieves stable policy improvement and consistently outperforms state-of-the-art RL methods on the OGBench offline RL benchmark and real-world robotic manipulation task. Videos are available on our website: https://agibottech.github.io/vine.
Rushuai Yang, Zhuo Han, Houlin Li +10
1AgiBot · 3Peking University · 2The Hong Kong University of Science and Technology
Pretrained generative robot policies based on flow matching and diffusion have achieved impressive results across a wide range of manipulation tasks. Yet real-world deployments routinely expose failure modes outside the pretraining distribution. Closing these gaps typically requires large-scale data collection or online reinforcement learning on physical hardware, which is impractical for rapid and safe adaptation. We present FlowDAgger, a sample- and compute-efficient method for adapting frozen generative robot policies from human interventions in latent space. Our key idea is action inversion: each human expert action is mapped to the noise that would have produced it under the frozen base policy, using reverse-time integration followed by local refinement. The resulting inverted noise provides supervision for a lightweight latent policy that steers the base model at deployment time, enabling rapid skill acquisition while preserving its behavioral priors. We evaluate FlowDAgger in simulation and on real-world bimanual and single-arm manipulation, adapting both action-head VLAs and world-action models from a handful of interventions. FlowDAgger outperforms supervised fine-tuning and latent-space RL baselines and preserves pretrained skills on held-out tasks, offering a practical path for adapting robot foundation models in the real world. Website: https://microsoft.github.io/FlowDAgger
Michael Murray, Daphne Chen, Simran Bagaria +7
1Microsoft Research · University of Washington · 2Microsoft +1
Offline reinforcement learning improves robotic policies using previously collected data without further environment interaction. Yet prevalent diffusion- and flow-matching robot policies lack tractable likelihoods, limiting their use in likelihood-based offline RL post-training. AR-NFs offer both expressive action modeling and exact likelihood evaluation, but their sequential sampling incurs substantial sampling overhead during policy optimization and deployment. We present RoMAN-Flow (Robotic Manipulation with Autoregressive Normalizing Flows), an offline reinforcement learning framework that makes AR-NF policies practical for robotic manipulation by addressing this sampling bottleneck in both stages. During policy optimization, RoMAN-Flow employs a sampling-free, advantage-weighted likelihood objective that assigns higher likelihood to high-advantage actions from the offline dataset without sampling from the autoregressive policy. For efficient deployment, it distills the optimized autoregressive policy into a one-step action generator, enabling low-latency action prediction. Experiments across multiple simulated manipulation benchmarks and real-world robotic platforms demonstrate that RoMAN-Flow achieves competitive policy performance while substantially reducing inference latency. Code is available at https://github.com/konnyaku28/RoMAN-Flow.
Shaoxuan Wang, Guangting Zheng, Rui Huang +4
University of Science and Technology of China · The Chinese University of Hong Kong