cs.ROMar 13, 2026

Beyond Imitation: Reinforcement Learning Fine-Tuning for Adaptive Diffusion Navigation Policies

Authors: Junhe Sheng, Ruofei Bai, Kuan Xu, Ruimeng Liu, Jie Chen, Shenghai Yuan, Wei-Yun Yau, Lihua Xie

Organizations: Nanyang Technological University, Singapore. · A*STAR, Singapore. · National University of Singapore, Singapore.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Diffusion-based robot navigation policies trained on large-scale imitation learning datasets, can generate multi-modal trajectories directly from the robot's visual observations, bypassing the traditional localization-mapping-planning pipeline and achieving strong zero-shot generalization. However, their performance remains constrained by the coverage of offline datasets, and when deployed in unseen settings, distribution shift often leads to accumulated trajectory errors and safety-critical failures. Adapting diffusion policies with reinforcement learning is challenging because their iterative denoising structure hinders effective gradient backpropagation, while also making the training of an additional value network computationally expensive and less stable. To address these issues, we propose a reinforcement learning fine-tuning framework tailored for diffusion-based navigation. The method leverages the inherent multi-trajectory sampling mechanism of diffusion models and adopts Group Relative Policy Optimization (GRPO), which estimates relative advantages across sampled trajectories without requiring a separate value network. To preserve pretrained representations while enabling adaptation, we freeze the visual encoder and selectively update the higher decoder layers and action head, enhancing safety-aware behaviors through online environmental feedback. On the PointGoal task in Isaac Sim, our approach improves the Success Rate from 52.0% to 58.7% and SPL from 0.49 to 0.54 on unseen scenes, while reducing collision frequency. Additional experiments show that the fine-tuned policy transfers zero-shot to a real quadruped platform and maintains stable performance in geometrically out-of-distribution environments, suggesting improved adaptability and safe generalization to new domains.

Figures & tables

Fig. 1: Overview of our framework.
ComponentDefinitionWeight
SuccessI(∥pH−g∥2<0.3)\mathbb{I}(\|p_{H}-g\|_{2}<0.3)+10.0
Progressmax⁡(0,∥p0−g∥2−∥pH−g∥2)\max(0,\|p_{0}-g\|_{2}-\|p_{H}-g\|_{2})+3.0
Collision1H∑h=1HI(ph∈Oocc)\frac{1}{H}\sum_{h=1}^{H}\mathbb{I}(p_{h}\in\mathcal{O}_{occ})-5.0
Inflated Coll.1H∑h=1HI(ph∈Oinfl)\frac{1}{H}\sum_{h=1}^{H}\mathbb{I}(p_{h}\in\mathcal{O}_{infl})-1.0
Distance∥pH−g∥2\|p_{H}-g\|_{2}-0.1
Smoothness1H−1∑h=1H−1∥ph+1−ph∥2\frac{1}{H{-}1}\sum_{h=1}^{H-1}\|p_{h+1}-p_{h}\|_{2}-0.1
TABLE I: Reward function components.
Fig. 2: Representative high-fidelity residential environments from the InternScenes-Home dataset used for training and evaluation. Image adapted from [ 3 ] .
EnvironmentMethodSuccess Rate ↑\uparrowSPL ↑\uparrow
SeenNavDP0.6010.6010.5770.577
SeenOurs0.687\mathbf{0.687}0.628\mathbf{0.628}
UnseenDD-PPO [ 8 ]0.1210.1210.3620.362
UnseeniPlanner [ 22 ]0.3620.3620.3450.345
UnseenViPlanner [ 23 ]0.4020.4020.3900.390
UnseenNavDP0.5200.5200.4900.490
TABLE II: Quantitative results on Seen (Training) and Unseen (Test) environments. Metrics are reported as the mean over 3 seeds.
Fig. 3: Trajectory comparison. Our fine-tuned policy (top row, green) generates tightly converged, safe avoidance bundles. The NavDP baseline (bottom row, red) produces highly divergent and unsafe paths.
Fig. 4: Zero-shot sim-to-sim transfer in Habitat. Our fine-tuned policy (top row) successfully navigates across diverse scenarios, while the NavDP baseline (bottom row) consistently fails.
Fine-Tuning StrategySuccess Rate ↑\uparrowSPL ↑\uparrow
Ours (RL-FT, FT5)0.587±0.014\mathbf{0.587\pm 0.014}0.537±0.002\mathbf{0.537\pm 0.002}
FT2 (2 layers)0.573±0.0140.573\pm 0.0140.534±0.0170.534\pm 0.017
FT-All (full model)0.520±0.0660.520\pm 0.0660.476±0.0730.476\pm 0.073
TABLE III: Effect of fine-tuning different model components.
Denoising Steps KKSuccess Rate ↑\uparrowSPL ↑\uparrow
K=3K=30.567±0.0200.567\pm 0.0200.517±0.0140.517\pm 0.014
K=5K=50.569±0.0090.569\pm 0.0090.520±0.0040.520\pm 0.004
K=7\mathbf{K=7}0.587±0.014\mathbf{0.587\pm 0.014}0.537±0.002\mathbf{0.537\pm 0.002}
K=10K=100.532±0.0150.532\pm 0.0150.490±0.0120.490\pm 0.012
TABLE IV: Effect of denoising steps KK on navigation performance.
MethodSuccess Rate ↑\uparrowSPL ↑\uparrow
Ours (RL-FT)0.587±0.014\mathbf{0.587\pm 0.014}0.537±0.002\mathbf{0.537\pm 0.002}
Ours w/o Clipping0.553±0.0140.553\pm 0.0140.500±0.0090.500\pm 0.009
Ours w/o Adv. Norm.0.565±0.0240.565\pm 0.0240.516±0.0180.516\pm 0.018
TABLE V: Ablation of GRPO objective components.
Fig. 5: Real-world navigation timelapse. Green and orange trajectories denote the actual executed paths of successful and failed runs, respectively. Our policy (top row) navigates smoothly, while the NavDP baseline (bottom row) struggles in dense clutter.
MethodSuccess Rate ↑\uparrow
NavDP (Pretrained)50% (10/20)
Ours (RL-FT)70% (14/20)
TABLE VI: Real-world navigation performance on Unitree Go2.
More in this topic

Robot Navigation

Robot navigation studies how robots perceive, localize, plan, and move safely toward goals in static or changing environments.

See all 554 papers

Explore similar work

CardsList
Jul 30, 2026

X-NavDP: Generalizing Navigation Diffusion Policy to Novel Behavior and Embodiments with Group Q-score Reweighted Matching

Pretraining navigation diffusion policies rely on large-scale expert demonstrations. These data are typically generated by a fully-informed oracle planner suited to a single nominal robot. This limits the policy's generalization to diverse embodiments and challenging scenarios (e.g., escaping dead ends or detouring long obstacles) that demand diverse local reactive behaviors with only onboard local observations. Post-training the policy with reinforcement learning (RL) offers a principled remedy. However, previous RL for diffusion approaches lead to only marginal improvements. This is because the intractable likelihood of diffusion policies renders policy gradients unstable in addition to inefficient policy exploration. To address these challenges, we propose a data-efficient diffusion RL post-training framework - GQRM (Group Q-score Reweighted Matching). Our framework introduces two complementary designs: (i) a self-bootstrapped exploration strategy with behavior perturbation that preserves the pretrained policy prior, and (ii) a group Q-score normalization mechanism that computes per-trajectory values on each state for efficient reweighted score matching. By conducting distributed online RL training across heterogeneous embodiments, the resulting fine-tuned policy, X-NavDP, achieves state-of-the-art cross-embodiment visual navigation performance, improving the overall success rate from 61.20% to 84.28% in simulation and 10% to 65% in real-world hard cases. The code and model are publicly available at https://yty-sky.github.io/x-navdp-project-page.

Tianyu Yang, Yiming Zeng, Wenzhe Cai +5

1Fudan University · 3Sun Yat-sen University · 2Shanghai AI Laboratory +1

Robot NavigationPolicy OptimizationCross-Embodiment Robot LearningVisual Robot NavigationDiffusion PolicyRL Post-Training+1
AskPDF
Sep 21, 2026

Performance-Preserving Online Adaptation in Social Navigation via Diffusion Steering

In social navigation, modeling the complex interactions between humans and robots is difficult, and deep reinforcement learning has therefore been actively studied. However, because simulation alone cannot fully reproduce diverse scenarios, robot dynamics, and the social conventions that vary across deployment environments, fine-tuning in the deployment environment is promising. In doing so, learning that preserves the base model's performance is required, so as not to compromise the primary objective of navigation, namely avoiding pedestrians and reaching the destination. In this study, we propose a method that applies diffusion steering via reinforcement learning (DSRL), which trains only the noise policy while keeping the diffusion policy fixed, thereby achieving learning that preserves performance. Furthermore, we integrate diffusion-based RL policies trained with multiple seeds to construct the base policy, improving learning performance. Our evaluation shows that, compared with other methods, the proposed method enables efficient learning while preserving performance, and we confirm flexible behavior control through adaptation to social conventions, as well as its effectiveness on a physical robot through hardware-in-the-loop simulation.

Haruto Nagahisa, Kohei Matsumoto, Yuki Hyodo +1

Graduate School of Information Science and Electrical Engineering, Kyushu University, Fukuoka, Japan · Faculty of Information Science and Electrical Engineering, Kyushu University, Fukuoka, Japan

Robot Policy AdaptationReinforcement LearningRobotic RLDiffusion Policy
AskPDF
Jun 9, 2026

MODIP: Efficient Model-Based Optimization for Diffusion Policies

Diffusion policies (DPs) have emerged as expressive policy representations for robot learning, often used with imitation learning methods such as behavioral cloning (BC). However, while their success has largely been confined to BC, direct reinforcement learning (RL) fine-tuning remains challenging because actions are generated through a multi-step denoising process. In this work, we propose MODIP, a framework for the offline-to-online fine-tuning of DPs. Rather than directly applying RL to the DPs, MODIP leverages a world model (WM) to guide policy adaptation and keeps the simplicity and stability of BC. We utilize model predictive control (MPC) to generate high-quality trajectories within the WM, and use them as supervised targets for fine-tuning the DP. To make MPC planning efficient, MODIP uses a terminal state value instead of a policy-dependent state-action value, reducing inference time. Additionally, MODIP trains critics with policy-independent TD targets, reducing training time. Experiments on D4RL (MuJoCo, Kitchen) and RoboMimic tasks show that MODIP improves diffusion policies beyond BC, and is competitive with or outperforms diffusion policy RL fine-tuning methods and strong model-based baselines such as TD-MPC2.

Zakariae El Asri, Philippe Gratias-Quiquandon, Nicolas Thome +1

Sorbonne Université, CNRS, ISIR, F-75005 Paris, France · Institut Universitaire de France (IUF)

Robot Policy LearningRobot Policy AdaptationDiffusion PolicyPolicy LearningOffline-To-Online RLModel-Based RL
AskPDF