Let's Reward Step-by-Step: Step-Aware Contrastive Alignment for Vision-Language Navigation in Continuous Environments

Haoyuan Li; Rui Liu; Hehe Fan; Yi Yang

cs.ROMar 10, 2026

Let's Reward Step-by-Step: Step-Aware Contrastive Alignment for Vision-Language Navigation in Continuous Environments

Authors: Haoyuan Li, Rui Liu, Hehe Fan, Yi Yang

Organizations: College of Computer Science and Technology, Zhejiang University, Hangzhou, China

Abstract

Vision-Language Navigation in Continuous Environments (VLN-CE) requires agents to learn complex reasoning from long-horizon human interactions. While Multi-modal Large Language Models (MLLMs) have driven recent progress, current training paradigms struggle to balance generalization capability, error recovery and training stability. Specifically, (i) policies derived from SFT suffer from compounding errors, struggling to recover from out-of-distribution states, and (ii) Reinforcement Fine-Tuning (RFT) methods e.g. GRPO are bottlenecked by sparse outcome rewards. Their binary feedback fails to assign credit to individual steps, leading to gradient signal collapse in failure dominant batches. To address these challenges, we introduce Step-Aware Contrastive Alignment (SACA), a framework designed to extract dense supervision from imperfect trajectories. At its core, the Perception-Grounded Step-Aware auditor evaluates progress step-by-step, disentangling failed trajectories into valid prefixes and exact divergence points. Leveraging these signals, Scenario-Conditioned Group Construction mechanism dynamically routes batches to specialized resampling and optimization strategies. Extensive experiments on VLN-CE benchmarks demonstrate that SACA achieves state-of-the-art performance.

Figures & tables

Figure 1: (a) Previous work discard entire trajectory upon failure due to compounding errors and sparse rewards. (b) SACA uses the PGSA auditor to pinpoint the exact divergence point ( ). Then uses Repair Resampling ( ) to recover from near-miss trajectories.

Figure 2: Overview of proposed SACA framework. The PGSA auditor evaluates $K$ trajectories against instruction landmarks, yielding a Soft Score for ranking and a Hard Mask to isolate the Divergence Point. Based on batch outcomes, a Scenario-Conditioned mechanism dynamically routes to either Repair Resampling (for mixed groups) or All-Failure Rescue (for null-outcome groups), followed by robust optimization.

Figure 3: Illustraion of the Repair Resampling process. (a) Extracting the structural mask $M_{t}$ via the PGSA auditor. (b) Backtracking to the Divergence Point $t_{div}$ to prune the erroneous suffix. (c) Resampling a corrective path from $t_{div}$ , thereby salvaging the valid prefix and providing robust step-level supervision.

Method	Observation Encoder				R2R-CE Val-Unseen				RxR-CE Val-Unseen
Method	Pano.	Odo.	Depth	S.RGB	NE $\downarrow$	OS $\uparrow$	SR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	SR $\uparrow$	SPL $\uparrow$	nDTW $\uparrow$
HPN+DN ∗ [ 24 ]	$\checkmark$	$\checkmark$	$\checkmark$		6.31	40.0	36.0	34.0	-	-	-	-
CMA ∗ [ 20 ]	$\checkmark$	$\checkmark$	$\checkmark$		6.20	52.0	41.0	36.0	8.76	26.5	22.1	47.0
VLN $\circlearrowright$ BERT ∗ [ 20 ]	$\checkmark$	$\checkmark$	$\checkmark$		5.74	53.0	44.0	39.0	8.98	27.0	22.6	46.7
Sim2Sim ∗ [ 25 ]	$\checkmark$	$\checkmark$	$\checkmark$		6.07	52.0	43.0	36.0	-	-	-	-
GridMM ∗ [ 53 ]	$\checkmark$	$\checkmark$	$\checkmark$		5.11	61.0	49.0	41.0	-	-	-	-

Table 1: Comparison with state-of-the-art methods.

*

indicates methods using the waypoint predictor from [ 20 ] .

\dagger

denotes methods using additional training data beyond the R2R-CE [ 2 ] and RxR-CE [ 27 ] benchmarks.

Figure 4: Qualitative comparison of navigation trajectories. (a) Egocentric observations. (b) Top-down maps. Green and red frames denote correct trajectory and failure trajectory, respectively.

Row	Components			R2R-CE Val-Unseen				RxR-CE Val-Unseen
Row	SS	AFR	RR	NE $\downarrow$	OS $\uparrow$	SR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	SR $\uparrow$	SPL $\uparrow$	nDTW $\uparrow$
0	SFT Baseline [ 55 ]			5.43	62.5	52.8	47.2	6.72	48.6	42.5	60.2
1				5.39	55.3	54.1	48.8	6.54	49.9	43.0	60.3
2	$\checkmark$			5.21	63.2	55.4	49.6	6.15	51.8	44.3	60.7
3	$\checkmark$	$\checkmark$		4.82	64.1	58.2	52.4	5.48	56.4	47.6	61.3
4	$\checkmark$	$\checkmark$	$\checkmark$	4.57	64.9	60.3	55.1	4.90	60.3	49.8	62.1

Table 2: Ablation study on the core components of SACA. SS : Soft Score for ranking; AFR : All-Failure Rescue (Scenario B); RR : Repair Resampling (Scenario A).

Figure 5: Illustration of SR curves during RFT training.

Figure 6: Qualitative visualization of the Perception-Grounded Step-Aware (PGSA) Auditor’s cascaded pipeline. (a) Raw continuous egocentric observations $o_{t}$ during navigation. (b) GroundingDINO [ 35 ] detects intermediate landmarks. (c) SAM3 [ 3 ] extracts precise pixel-level masks. (d) The corresponding top-down trajectory mapping.

Ablation on Objective Constraints
Model Variant	R2R-CE Val-Unseen				RxR-CE Val-Unseen
Model Variant	NE $\downarrow$	OS $\uparrow$	SR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	SR $\uparrow$	SPL $\uparrow$	nDTW $\uparrow$
SACA	4.57	64.9	60.3	55.1	4.90	60.3	49.8	62.1
w/o Consistency Align ( $\mathcal{L}_{align}$ )	4.89	63.8	58.0	51.6	5.34	57.1	46.2	60.5
w/o Contrastive Correct ( $\mathcal{L}_{correct}$ )	4.98	63.5	57.3	52.8	5.51	56.4	46.5	59.4
Ablation on Robustness Mechanisms

Table 3: Ablation study on specific objective designs and robustness mechanisms within SACA. All variants are compared against the full SACA model.

Group Size ( $K$ )	Max Repairs ( $N_{rep}$ )	R2R-CE Val-Unseen				RxR-CE Val-Unseen
Group Size ( $K$ )	Max Repairs ( $N_{rep}$ )	NE $\downarrow$	OS $\uparrow$	SR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	SR $\uparrow$	SPL $\uparrow$	nDTW $\uparrow$
4	3	4.92	62.1	56.4	51.2	5.30	55.2	45.3	59.5
8	3	4.57	64.9	60.3	55.1	4.90	60.3	49.8	62.1
16	3	4.59	64.7	60.1	54.8	4.93	59.9	49.5	61.8
8	1	4.81	63.5	58.1	53.4	5.15	57.5	47.1	60.2
8	5	4.60	64.8	60.0	54.9	4.92	60.1	49.6	62.0

Table 4: Ablation study on the group size (

K

) and maximum repair attempts (

N_{rep}

Row	Perception Modules			R2R-CE Val-Unseen				RxR-CE Val-Unseen
Row	Global	BBox	Mask	NE $\downarrow$	OS $\uparrow$	SR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	SR $\uparrow$	SPL $\uparrow$	nDTW $\uparrow$
1	$\checkmark$			4.95	63.2	57.6	52.1	5.45	56.5	46.2	59.5
2	$\checkmark$	$\checkmark$		4.71	64.1	59.2	54.0	5.11	58.7	48.3	61.1
3	$\checkmark$	$\checkmark$	$\checkmark$	4.57	64.9	60.3	55.1	4.90	60.3	49.8	62.1

Table 5: Ablation study on the perception modules within the PGSA Auditor. Global : CLIP [ 41 ] Image-Text Similarity; BBox : GroundingDINO [ 35 ] object detection; Mask : SAM3 [ 3 ] precise segmentation.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1 : Distribution of valid-prefix ratios among failed navigation trajectories.

Trajectory Phase	Temporal Condition	Hard Mask	Dominant Reward Signal ( $\hat{A}_{t}$ )	Optimization Objective
Valid Prefix	$t<t_{div}$	$M_{t}=1$	Dense Positive (Continuous Baseline + Peaks)	$\mathcal{L}_{align}$ (Behavior Cloning)
Divergence Point	$t=t_{div}$	$M_{t}=0$	Sharp Negative (Explicit Penalty)	$\mathcal{L}_{corr}$ (Contrastive Correction)
Resampled Suffix	$t>t_{div}$	$M_{t}=1$	Dense Positive (Continuous Baseline + Peaks)	$\mathcal{L}_{repair}$ (Suffix Cloning)

Appendix

Table S1 : Summary of Step-Level Reward and Objective Allocation in SACA.

Figure S2 : Heatmap visualization. We visualize the step-wise evolution of semantic alignment and advantage assignment over a trajectory consist with $48$ steps.

Panel of Figure 15 — Algorithm 1 Perception-Grounded Step-Aware (PGSA) Auditing Process

Panel of Figure 16 — Algorithm 2 Step-Aware Contrastive Alignment (SACA) Framework

General & SFT Configuration		SACA Hyperparameters
Hyperparameter	Value	Hyperparameter	Value
General Infrastructure		Scenario & Grouping
Policy Model	Video-LLaVA-8B	Group size ( $K$ )	8
Max Sequence Length	4096	Valid-prefix ratio ( $\eta$ )	0.5
Number of GPUs	8	Max resampling attempts ( $N_{rep}$ )	3
Parallel Strategy	DDP	PGSA Auditor Settings

Appendix

Table S2 : Comprehensive training configurations for SFT and SACA stages.

Figure S3 : Qualitative visualization of navigation episodes. (a) Continuous egocentric visual observations. (b) The corresponding top-down trajectory maps. Key textual landmarks parsed from the instructions are highlighted in red, guiding the step-aware progress tracking.

Figure S4 : More qualitative comparison. (a) Egocentric observations. (b) Top-down maps. Green and red frames denote correct trajectory and failure trajectory, respectively.

Vision-Language Navigation

Vision-Language Navigation studies how embodied agents use natural-language instructions and visual observations to plan and execute routes.

See all 310 papers

SeeNav-Agent: Enhancing Vision-Language Navigation with Visual Prompt and Step-Level Policy Optimization
Dec 2, 2025Zhengcheng Wang, Zichuan Lin, Yijun Yang +2Vision-Language Navigation
Tencent AI Lab
ReflectVLN: Training Vision-Language Navigation Agents with Reflective Reasoning
Jul 14, 2026Jiahang Wang, Yirong Yang, Yanqing Zhu +4Hierarchical Planning Vision-Language Navigation
Amap, Alibaba Group, China · Beihang University
Dual-Anchoring: Addressing State Drift in Vision-Language Navigation
Apr 19, 2026Kangyi Wu, Pengna Li, Kailin Lyu +5VLM Adaptation Vision-Language Navigation
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, National Engineering Research Center for Visual Information and Applications, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · Institute of Automation, Chinese Academy of Sciences, School of Artificial Intelligence, University of Chinese Academy of Sciences · Johns Hopkins University +2

Dec 2, 2025J/K move · Enter open · S save

SeeNav-Agent: Enhancing Vision-Language Navigation with Visual Prompt and Step-Level Policy Optimization

Zhengcheng Wang, Zichuan Lin, Yijun Yang, Haobo Fu+1

Tencent AI Lab

Vision-Language Navigation

Open paper AskPDF arXiv

Existing Vision-Language Navigation (VLN) agents based on Large Vision-Language Models (LVLMs) often suffer from errors of perception, reasoning, and planning, which significantly hinder their navigation performance. To address these limitations, a novel VLN agent framework, named SeeNav-Agent, is proposed. First, to reduce hallucinations of the visual module of the VLN agent, a dual-view Visual Prompt (VP) technique is introduced, which can also improve the agent's understanding of current spatial states. Subsequently, a novel step-level Reinforcement Fine-Tuning (RFT) method, Step Reward Group Policy Optimization (SRGPO), is designed for post-training. In SRGPO, we first define verifiable process rewards for navigation, and then perform efficient step-level advantage estimation by randomly grouping navigation steps. SRGPO provides dense reward signals for the RFT of the VLN agent and enhances its planning capability. Experiments on both the EmbodiedBench Navigation (Embench-Nav) and VLN-CE validate the effectiveness of the proposed method. By introducing the zero-shot VP module, the GPT-4.1 achieves a navigation success rate of 86.7%, surpassing the current best LVLM by approximately 20 percentage points (pp) on Embench-Nav and achieves an 11.1 pp improvement on VLN-CE compared to the base model. With VP+SRGPO, the Qwen2.5-VL-3B model reaches a navigation success rate of 72.3%, outperforming the best existing LVLM model by 5.6 pp on Embench-Nav and achieves 8.9 pp improvement on VLN-CE compared to the baseline method. Moreover, compared to GRPO and GiGPO, the proposed SRGPO demonstrates significant improvements in training stability, convergence efficiency, and generalization capability.

Abstract

Vision-Language Navigation

Explore similar work

SeeNav-Agent: Enhancing Vision-Language Navigation with Visual Prompt and Step-Level Policy Optimization

ReflectVLN: Training Vision-Language Navigation Agents with Reflective Reasoning

Dual-Anchoring: Addressing State Drift in Vision-Language Navigation