Organizations: School of Software Engineering, South China University of Technology, Guangzhou, China · Institute for Super Robotics (Huangpu), Guangzhou, China · Key Laboratory of Big Data and Intelligent Robot, Ministry of Education, Guangzhou 510006, China
Vision-and-Language Navigation (VLN) requires agents to follow instructions and navigate to target locations. However, current VLN agents often suffer severe performance degradation in unseen environments due to distribution shifts. Test-Time Adaptation (TTA) offers a promising paradigm to bridge this gap, yet existing methods that rely on unsupervised entropy minimization can propagate errors through the trajectory, causing cascading failures in navigation. Recent methods mitigate this by incorporating human feedback, but they apply trajectory-level supervision uniformly across all steps without localizing the actions is truly informative, leading to inefficient feedback utilization and mismatched optimization granularity. To address these limitations, we propose FETA, a Feedback-Efficient Test-Time Adaptation method that improves both feedback efficiency and optimization reliability. To reduce unnecessary human interventions, we introduce a hierarchical sample selection strategy to localize informative samples. Specifically, we first filter trajectories whose normalized policy entropy exhibits large fluctuations. Within each selected trajectory, we further select the critical action via an entropy-gradient saliency criterion and query a single binary feedback on its correctness. A discrepancy-aware entropy optimization objective then updates the policy only at the queried step, with adaptive scaling that assigns larger relative weights to samples with greater prediction-feedback discrepancy. Extensive experiments across multiple benchmarks demonstrate that FETA achieves state-of-the-art performance with fewer human interventions, while remaining robust under noisy feedback and continual adaptation settings.
Figures & tables
Algorithm 1 The pipeline of F eedback- E fficient T est-Time A daptation method for VLN
Fig. 1: Overview of FETA. For the trajectory executed by the agent, we hierarchically select informative samples by filtering trajectories with higher normalized-entropy standard deviation σH^(τ) and querying the step with the largest entropy-gradient saliency Gt . The queried feedback is then used to update the policy via directional entropy optimization modulated by discrepancy-aware scaling, enabling efficient adaptation in novel environments.
Figure 3
Methods
Val Seen
Val Unseen
Time(ms)
OSR ↑
SR ↑
SPL ↑
RGSPL ↑
FC ↓
OSR ↑
SR ↑
SPL ↑
RGSPL ↑
FC ↓
DUET [ 3 ]
73.86
71.75
63.94
51.14
–
51.07
46.98
33.73
23.03
–
156.5
• FSTTA [ 6 ]
73.79
71.68
64.00
51.21
–
50.93
46.86
33.71
23.00
–
457.7
• FeedTTA [ 20 ]
82.83
77.28
65.39
54.28
1424
62.60
53.85
28.66
19.02
3522
433.1
• ATENA [ 7 ]
82.62
81.40
72.85
58.90
573
65.78
61.95
42.73
28.39
2595
321.3
• FETA
84.31
82.29
73.75
59.01
519
73.27
68.33
50.55
33.15
2475
243.5
TABLE I: Experimental results on REVERIE. The last column reports the average runtime per episode.
Methods
Test Residential Basic
Test Non-residential Scene
SR ↑
SPL ↑
NDTW ↑
NE ↓
SR ↑
SPL ↑
NDTW ↑
NE ↓
GR-DUET [ 19 ]
69
64
71
3.09
47
42
54
5.55
• FSTTA [ 6 ]
69
62
70
3.14
48
42
53
5.46
• FeedTTA [ 20 ]
69
64
72
3.07
48
43
55
5.43
• ATENA [ 7 ]
67
64
71
3.34
48
40
50
5.45
• FETA
71
67
74
2.93
50
44
55
4.94
TABLE II: Experimental results on GSA-R2R.
Methods
Val Seen
Val Unseen
OSR ↑
SR ↑
SPL ↑
NE ↓
OSR ↑
SR ↑
SPL ↑
NE ↓
ETPNav [ 10 ]
71
64
57
4.12
64
57
49
4.72
• FSTTA [ 6 ]
70
63
56
4.19
63
55
47
4.86
• FeedTTA [ 20 ]
72
65
58
4.06
64
57
49
4.73
• ATENA [ 7 ]
70
65
59
4.06
63
57
50
4.69
• FETA
72
66
59
4.00
65
59
51
4.52
TABLE III: Experimental results on R2R-CE.
Module
Val Unseen
TLS
SLS
DAS
SR ↑
OSR ↑
SPL ↑
RGSPL ↑
RGS ↑
Baseline
46.98
51.07
33.73
23.03
32.15
✓
-
-
55.54
64.21
36.10
24.12
37.06
-
✓
-
61.41
69.01
43.32
29.14
41.47
-
-
✓
56.83
62.52
37.63
25.11
37.91
-
✓
✓
63.19
67.52
46.86
31.32
42.34
TABLE IV: Ablation study of Trajectory-Level Selection (TLS), Step-Level Selection (SLS), and Discrepancy-Aware Scaling (DAS) on the REVERIE Val Unseen split. Baseline denotes the unadapted model.
Data (Train → Test)
Methods
Val Unseen
SR ↑
SPL ↑
NE ↓
TL ↓
R2R → REVERIE
DUET [ 3 ]
24.91
20.00
7.22
14.67
• ATENA
29.42
19.43
6.75
24.45
• FETA
40.02
23.53
5.20
39.09
REVERIE → R2R
DUET [ 3 ]
29.84
9.03
7.28
44.20
• ATENA
31.25
8.41
6.98
48.40
TABLE V: Experimental results of cross-dataset adaptation. Models are trained on one dataset and adapted on the other.
Method
Office
Laboratory
SR ↑
NE ↓
SR ↑
NE ↓
BEVBert [ 12 ]
40
4.30
30
4.20
• FETA
55
3.87
40
4.03
TABLE VI: Experimental results in real-world environments.
Selection Method
Val Seen
Val Unseen
OSR ↑
SR ↑
SPL ↑
RGSPL ↑
OSR ↑
SR ↑
SPL ↑
RGSPL ↑
Entropy
81.19
79.03
71.31
57.41
69.63
63.97
46.39
31.13
Entropy-Gradient Saliency
84.31
82.29
73.75
59.01
73.27
68.33
50.55
33.15
TABLE VII: Comparison of step-level selection methods on the REVERIE dataset.
Fig. 4: Effect of noisy binary feedback on navigation performance in the REVERIE Val Unseen split.
Fig. 5: Continual adaptation results on REVERIE Unseen split, averaged over 3 random seeds.
Fig. 6: Qualitative comparison of navigation trajectories between the base model and our test-time adaptation method (FETA). Red waypoints indicate the trajectory predicted by the base model, while green waypoints denote the trajectory predicted after applying FETA at test time. Blue waypoints are the shared prefix visited by both methods before their trajectories diverge.
Fig. 7: Real-robot navigation examples. (a) BEVBert fails to reach the blue display board, while FETA proceeds past the sofas and stops near the board. (b) BEVBert remains near the starting location, while FETA successfully stops in front of the glass door.
Fig. 8: Human annotation interface for binary action feedback. Annotators assess action usefulness using the task instruction, trajectory video, and panoramic observations before and after the queried action.
Test-time adaptation for vision-language navigation (TTA-VLN) enables pretrained policies to adapt online to unseen environments using only test-time observations and interaction history. However, distribution shifts can distort local action preferences and lead to off-course decisions. Existing methods rely on predictive uncertainty, trajectory-level feedback, or accumulated adaptation experience to correct such deviations. These signals, however, do not directly reveal whether an executed action supports instruction-guided progress toward the goal. Moreover, a plausible corrective signal does not guarantee a reliable policy update. The key challenge is thus twofold: identifying interactions that support goal-directed improvement and determining whether the resulting updates are worth retaining. We observe that each executed action induces an immediate observation transition, providing evidence of its local consequences. Based on this insight, we propose Credit-Guided Policy Improvement (CGPI), which recovers signed, reference-relative decision credit from action-induced observation transitions without external outcome feedback. With the pretrained navigation policy frozen, CGPI uses this credit to propose lightweight adaptation updates and verifies them against prior credit-supported interactions. Updates are retained only when supported and rolled back otherwise. CGPI achieves consistent gains across the evaluated VLN benchmarks and navigation backbones, while qualitative robot trials further illustrate the feasibility of zero-shot sim-to-real transfer.
Yang Li, Sijia Zhang, Yihan Li +4
School of Artificial Intelligence, Tianjin University, China · The University of Hong Kong, Hong Kong SAR, China · School of Computer Science and Information Engineering, Hefei University of Technology, China
Test-time training (TTT) offers a lightweight way to adapt vision--language--action (VLA) policies from unlabeled deployment streams, but it remains difficult to use reliably in closed-loop manipulation. A shared adaptation space can mix incompatible task corrections, while an online update can alter subsequent actions before its consequences are known. We introduce a reliable TTT framework for VLA policies (VANE). VANE conditions prompt adaptation on the current vision--language context and learns from the future visual consequences of executed actions. Candidate updates are isolated from the live policy, evaluated on subsequent observations, and committed only when supported by future evidence, making adaptation selective and reversible. On SimplerEnv WidowX, VANE improves average success by 3.2 percentage points over the corresponding TTT baseline. Results on Google Robot further show that deployment-time gains remain task- and embodiment-dependent. Together, these results demonstrate a constrained, evidence-based approach to adapting VLA policies during interaction.
Hongjin Ji, Guoyang Xia, Luoyang Sun +2
The Chinese University of Hong Kong, Shenzhen · Li Auto Inc. · Beijing University of Posts and Telecommunications +2
Deploying vision-and-language navigation (VLN) agents requires adaptation across diverse scenes and environments, but fine-tuning on a specific scenario often causes catastrophic forgetting in others, which severely limits flexible long-term deployment. We formalize this challenge as the all-day multi-scenes lifelong VLN (AML-VLN) problem. Existing parameter-efficient adapters (e.g., LoRA and its variants) are limited by their two-dimensional matrix form, which fails to capture the multi-hierarchical navigation knowledge spanning multiple scenes and environments. To address this, we propose Tucker Adaptation (TuKA), which represents the multi-hierarchical navigation knowledge as a high-order tensor and leverages Tucker decomposition to decouple the knowledge into shared subspaces and scenario-specific experts. We further introduce a decoupled knowledge incremental learning strategy to consolidate shared subspaces while constraining specific experts for decoupled lifelong learning. Building on TuKA, we also develop a VLN agent named AlldayWalker, which continually learns across multiple navigation scenarios, achieving all-day multi-scenes navigation. Extensive experiments show that AlldayWalker consistently outperforms state-of-the-art baselines.
Xudong Wang, Gan Li, Zhiyu Liu +3
State Key Laboratory of Robotics and Intelligent Systems, Shenyang Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · North University of China +1