Asking Less, Adapting Better: Feedback-Efficient Test-Time Adaptation for Vision-and-Language Navigation

Zixu Wang; Huaquan Zhang; Shoukai Xu; Jinwu Hu; Zhiquan Wen; Mingkui Tan

cs.CVOct 22, 2026

Asking Less, Adapting Better: Feedback-Efficient Test-Time Adaptation for Vision-and-Language Navigation

Authors: Zixu Wang, Huaquan Zhang, Shoukai Xu, Jinwu Hu, Zhiquan Wen, Mingkui Tan

Organizations: School of Software Engineering, South China University of Technology, Guangzhou, China · Institute for Super Robotics (Huangpu), Guangzhou, China · Key Laboratory of Big Data and Intelligent Robot, Ministry of Education, Guangzhou 510006, China

Abstract

Vision-and-Language Navigation (VLN) requires agents to follow instructions and navigate to target locations. However, current VLN agents often suffer severe performance degradation in unseen environments due to distribution shifts. Test-Time Adaptation (TTA) offers a promising paradigm to bridge this gap, yet existing methods that rely on unsupervised entropy minimization can propagate errors through the trajectory, causing cascading failures in navigation. Recent methods mitigate this by incorporating human feedback, but they apply trajectory-level supervision uniformly across all steps without localizing the actions is truly informative, leading to inefficient feedback utilization and mismatched optimization granularity. To address these limitations, we propose FETA, a Feedback-Efficient Test-Time Adaptation method that improves both feedback efficiency and optimization reliability. To reduce unnecessary human interventions, we introduce a hierarchical sample selection strategy to localize informative samples. Specifically, we first filter trajectories whose normalized policy entropy exhibits large fluctuations. Within each selected trajectory, we further select the critical action via an entropy-gradient saliency criterion and query a single binary feedback on its correctness. A discrepancy-aware entropy optimization objective then updates the policy only at the queried step, with adaptive scaling that assigns larger relative weights to samples with greater prediction-feedback discrepancy. Extensive experiments across multiple benchmarks demonstrate that FETA achieves state-of-the-art performance with fewer human interventions, while remaining robust under noisy feedback and continual adaptation settings.

Figures & tables

Panel of Figure 1 — Algorithm 1 The pipeline of F eedback- E fficient T est-Time A daptation method for VLN

Fig. 1: Overview of FETA. For the trajectory executed by the agent, we hierarchically select informative samples by filtering trajectories with higher normalized-entropy standard deviation $\sigma_{\hat{H}}(\tau)$ and querying the step with the largest entropy-gradient saliency $G_{t}$ . The queried feedback is then used to update the policy via directional entropy optimization modulated by discrepancy-aware scaling, enabling efficient adaptation in novel environments.

Methods	Val Seen					Val Unseen					Time(ms)
Methods	OSR $\uparrow$	SR $\uparrow$	SPL $\uparrow$	RGSPL $\uparrow$	FC $\downarrow$	OSR $\uparrow$	SR $\uparrow$	SPL $\uparrow$	RGSPL $\uparrow$	FC $\downarrow$	Time(ms)
DUET [ 3 ]	73.86	71.75	63.94	51.14	–	51.07	46.98	33.73	23.03	–	156.5
• FSTTA [ 6 ]	73.79	71.68	64.00	51.21	–	50.93	46.86	33.71	23.00	–	457.7
• FeedTTA [ 20 ]	82.83	77.28	65.39	54.28	1424	62.60	53.85	28.66	19.02	3522	433.1
• ATENA [ 7 ]	82.62	81.40	72.85	58.90	573	65.78	61.95	42.73	28.39	2595	321.3
• FETA	84.31	82.29	73.75	59.01	519	73.27	68.33	50.55	33.15	2475	243.5

TABLE I: Experimental results on REVERIE. The last column reports the average runtime per episode.

Methods	Test Residential Basic				Test Non-residential Scene
Methods	SR $\uparrow$	SPL $\uparrow$	NDTW $\uparrow$	NE $\downarrow$	SR $\uparrow$	SPL $\uparrow$	NDTW $\uparrow$	NE $\downarrow$
GR-DUET [ 19 ]	69	64	71	3.09	47	42	54	5.55
• FSTTA [ 6 ]	69	62	70	3.14	48	42	53	5.46
• FeedTTA [ 20 ]	69	64	72	3.07	48	43	55	5.43
• ATENA [ 7 ]	67	64	71	3.34	48	40	50	5.45
• FETA	71	67	74	2.93	50	44	55	4.94

TABLE II: Experimental results on GSA-R2R.

Methods	Val Seen				Val Unseen
Methods	OSR $\uparrow$	SR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	OSR $\uparrow$	SR $\uparrow$	SPL $\uparrow$	NE $\downarrow$
ETPNav [ 10 ]	71	64	57	4.12	64	57	49	4.72
• FSTTA [ 6 ]	70	63	56	4.19	63	55	47	4.86
• FeedTTA [ 20 ]	72	65	58	4.06	64	57	49	4.73
• ATENA [ 7 ]	70	65	59	4.06	63	57	50	4.69
• FETA	72	66	59	4.00	65	59	51	4.52

TABLE III: Experimental results on R2R-CE.

Module			Val Unseen
TLS	SLS	DAS	SR $\uparrow$	OSR $\uparrow$	SPL $\uparrow$	RGSPL $\uparrow$	RGS $\uparrow$
Baseline			46.98	51.07	33.73	23.03	32.15
✓	-	-	55.54	64.21	36.10	24.12	37.06
-	✓	-	61.41	69.01	43.32	29.14	41.47
-	-	✓	56.83	62.52	37.63	25.11	37.91
-	✓	✓	63.19	67.52	46.86	31.32	42.34

TABLE IV: Ablation study of Trajectory-Level Selection (TLS), Step-Level Selection (SLS), and Discrepancy-Aware Scaling (DAS) on the REVERIE Val Unseen split. Baseline denotes the unadapted model.

Data (Train $\rightarrow$ Test)	Methods	Val Unseen
Data (Train $\rightarrow$ Test)	Methods	SR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	TL $\downarrow$
R2R $\rightarrow$ REVERIE	DUET [ 3 ]	24.91	20.00	7.22	14.67
	• ATENA	29.42	19.43	6.75	24.45
	• FETA	40.02	23.53	5.20	39.09
REVERIE $\rightarrow$ R2R	DUET [ 3 ]	29.84	9.03	7.28	44.20
	• ATENA	31.25	8.41	6.98	48.40

TABLE V: Experimental results of cross-dataset adaptation. Models are trained on one dataset and adapted on the other.

Method	Office		Laboratory
Method	SR $\uparrow$	NE $\downarrow$	SR $\uparrow$	NE $\downarrow$
BEVBert [ 12 ]	40	4.30	30	4.20
• FETA	55	3.87	40	4.03

TABLE VI: Experimental results in real-world environments.

Selection Method	Val Seen				Val Unseen
Selection Method	OSR $\uparrow$	SR $\uparrow$	SPL $\uparrow$	RGSPL $\uparrow$	OSR $\uparrow$	SR $\uparrow$	SPL $\uparrow$	RGSPL $\uparrow$
Entropy	81.19	79.03	71.31	57.41	69.63	63.97	46.39	31.13
Entropy-Gradient Saliency	84.31	82.29	73.75	59.01	73.27	68.33	50.55	33.15

TABLE VII: Comparison of step-level selection methods on the REVERIE dataset.

Fig. 4: Effect of noisy binary feedback on navigation performance in the REVERIE Val Unseen split.

Fig. 5: Continual adaptation results on REVERIE Unseen split, averaged over 3 random seeds.

Panel of Figure 13 — Fig. 6: Qualitative comparison of navigation trajectories between the base model and our test-time adaptation method (FETA). Red waypoints indicate the trajectory predicted by the base model, while green waypoints denote the trajectory predicted after applying FETA at test time. Blue waypoints are the shared prefix visited by both methods before their trajectories diverge.

Fig. 7: Real-robot navigation examples. (a) BEVBert fails to reach the blue display board, while FETA proceeds past the sofas and stops near the board. (b) BEVBert remains near the starting location, while FETA successfully stops in front of the glass door.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Methods	Val Seen				Val Unseen
Methods	OSR $\uparrow$	SR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	OSR $\uparrow$	SR $\uparrow$	SPL $\uparrow$	NE $\downarrow$
DUET [ 3 ]	86	79	73	2.28	81	72	60	3.31
• FSTTA [ 6 ]	83	76	69	2.51	78	68	57	3.47
• FeedTTA [ 20 ]	85	78	73	2.29	80	72	61	3.26
• ATENA [ 7 ]	85	79	74	2.28	79	72	62	3.23
• FETA	86	80	74	2.20	82	74	62	3.01

Appendix

TABLE VIII: Experimental results on R2R dataset.

Fig. 8: Human annotation interface for binary action feedback. Annotators assess action usefulness using the task instruction, trajectory video, and panoramic observations before and after the queried action.

Vision-Language Navigation

Vision-Language Navigation studies how embodied agents use natural-language instructions and visual observations to plan and execute routes.

See all 310 papers

Credit-Guided Policy Improvement for Test-time Adaptive Vision-Language Navigation
Sep 29, 2026Yang Li, Sijia Zhang, Yihan Li +4Robot Policy Learning Robot Policy Adaptation
School of Artificial Intelligence, Tianjin University, China · The University of Hong Kong, Hong Kong SAR, China · School of Computer Science and Information Engineering, Hefei University of Technology, China
VANE: Reliable Test-Time Training for Vision-Language-Action Models via Future Visual Representation Prediction
Aug 10, 2026Hongjin Ji, Guoyang Xia, Luoyang Sun +2Robot Policy Adaptation Vision-Language-Action Models
The Chinese University of Hong Kong, Shenzhen · Li Auto Inc. · Beijing University of Posts and Telecommunications +2
All-day Multi-scenes Lifelong Vision-and-Language Navigation with Tucker Adaptation
Mar 15, 2026Xudong Wang, Gan Li, Zhiyu Liu +3Tucker Decomposition Vision-Language Navigation
State Key Laboratory of Robotics and Intelligent Systems, Shenyang Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · North University of China +1

Sep 29, 2026J/K move · Enter open · S save

Credit-Guided Policy Improvement for Test-time Adaptive Vision-Language Navigation

Yang Li, Sijia Zhang, Yihan Li, Aming WU+3

School of Artificial Intelligence, Tianjin University, China · The University of Hong Kong, Hong Kong SAR, China · School of Computer Science and Information Engineering, Hefei University of Technology, China

Robot Policy Learning Robot Policy Adaptation Vision-Language Navigation Credit Assignment

Open paper AskPDF arXiv

Test-time adaptation for vision-language navigation (TTA-VLN) enables pretrained policies to adapt online to unseen environments using only test-time observations and interaction history. However, distribution shifts can distort local action preferences and lead to off-course decisions. Existing methods rely on predictive uncertainty, trajectory-level feedback, or accumulated adaptation experience to correct such deviations. These signals, however, do not directly reveal whether an executed action supports instruction-guided progress toward the goal. Moreover, a plausible corrective signal does not guarantee a reliable policy update. The key challenge is thus twofold: identifying interactions that support goal-directed improvement and determining whether the resulting updates are worth retaining. We observe that each executed action induces an immediate observation transition, providing evidence of its local consequences. Based on this insight, we propose Credit-Guided Policy Improvement (CGPI), which recovers signed, reference-relative decision credit from action-induced observation transitions without external outcome feedback. With the pretrained navigation policy frozen, CGPI uses this credit to propose lightweight adaptation updates and verifies them against prior credit-supported interactions. Updates are retained only when supported and rolled back otherwise. CGPI achieves consistent gains across the evaluated VLN benchmarks and navigation backbones, while qualitative robot trials further illustrate the feasibility of zero-shot sim-to-real transfer.

Abstract

Vision-Language Navigation

Explore similar work

Credit-Guided Policy Improvement for Test-time Adaptive Vision-Language Navigation

VANE: Reliable Test-Time Training for Vision-Language-Action Models via Future Visual Representation Prediction

All-day Multi-scenes Lifelong Vision-and-Language Navigation with Tucker Adaptation