Diffusion-based robot navigation policies trained on large-scale imitation learning datasets, can generate multi-modal trajectories directly from the robot's visual observations, bypassing the traditional localization-mapping-planning pipeline and achieving strong zero-shot generalization. However, their performance remains constrained by the coverage of offline datasets, and when deployed in unseen settings, distribution shift often leads to accumulated trajectory errors and safety-critical failures. Adapting diffusion policies with reinforcement learning is challenging because their iterative denoising structure hinders effective gradient backpropagation, while also making the training of an additional value network computationally expensive and less stable. To address these issues, we propose a reinforcement learning fine-tuning framework tailored for diffusion-based navigation. The method leverages the inherent multi-trajectory sampling mechanism of diffusion models and adopts Group Relative Policy Optimization (GRPO), which estimates relative advantages across sampled trajectories without requiring a separate value network. To preserve pretrained representations while enabling adaptation, we freeze the visual encoder and selectively update the higher decoder layers and action head, enhancing safety-aware behaviors through online environmental feedback. On the PointGoal task in Isaac Sim, our approach improves the Success Rate from 52.0% to 58.7% and SPL from 0.49 to 0.54 on unseen scenes, while reducing collision frequency. Additional experiments show that the fine-tuned policy transfers zero-shot to a real quadruped platform and maintains stable performance in geometrically out-of-distribution environments, suggesting improved adaptability and safe generalization to new domains.
Figures & tables
Fig. 1: Overview of our framework.
Component
Definition
Weight
Success
I(∥pH−g∥2<0.3)
+10.0
Progress
max(0,∥p0−g∥2−∥pH−g∥2)
+3.0
Collision
H1∑h=1HI(ph∈Oocc)
-5.0
Inflated Coll.
H1∑h=1HI(ph∈Oinfl)
-1.0
Distance
∥pH−g∥2
-0.1
Smoothness
H−11∑h=1H−1∥ph+1−ph∥2
-0.1
TABLE I: Reward function components.
Fig. 2: Representative high-fidelity residential environments from the InternScenes-Home dataset used for training and evaluation. Image adapted from [ 3 ] .
Environment
Method
Success Rate ↑
SPL ↑
Seen
NavDP
0.601
0.577
Seen
Ours
0.687
0.628
Unseen
DD-PPO [ 8 ]
0.121
0.362
Unseen
iPlanner [ 22 ]
0.362
0.345
Unseen
ViPlanner [ 23 ]
0.402
0.390
Unseen
NavDP
0.520
0.490
TABLE II: Quantitative results on Seen (Training) and Unseen (Test) environments. Metrics are reported as the mean over 3 seeds.
Fig. 4: Zero-shot sim-to-sim transfer in Habitat. Our fine-tuned policy (top row) successfully navigates across diverse scenarios, while the NavDP baseline (bottom row) consistently fails.
Fine-Tuning Strategy
Success Rate ↑
SPL ↑
Ours (RL-FT, FT5)
0.587±0.014
0.537±0.002
FT2 (2 layers)
0.573±0.014
0.534±0.017
FT-All (full model)
0.520±0.066
0.476±0.073
TABLE III: Effect of fine-tuning different model components.
Denoising Steps K
Success Rate ↑
SPL ↑
K=3
0.567±0.020
0.517±0.014
K=5
0.569±0.009
0.520±0.004
K=7
0.587±0.014
0.537±0.002
K=10
0.532±0.015
0.490±0.012
TABLE IV: Effect of denoising steps K on navigation performance.
Method
Success Rate ↑
SPL ↑
Ours (RL-FT)
0.587±0.014
0.537±0.002
Ours w/o Clipping
0.553±0.014
0.500±0.009
Ours w/o Adv. Norm.
0.565±0.024
0.516±0.018
TABLE V: Ablation of GRPO objective components.
Fig. 5: Real-world navigation timelapse. Green and orange trajectories denote the actual executed paths of successful and failed runs, respectively. Our policy (top row) navigates smoothly, while the NavDP baseline (bottom row) struggles in dense clutter.
Method
Success Rate ↑
NavDP (Pretrained)
50% (10/20)
Ours (RL-FT)
70% (14/20)
TABLE VI: Real-world navigation performance on Unitree Go2.
Graduate School of Information Science and Electrical Engineering, Kyushu University, Fukuoka, Japan · Faculty of Information Science and Electrical Engineering, Kyushu University, Fukuoka, Japan