MOON: Learning When to Forget and How to Move On in Vision-and-Language Navigation

Jingmeng Zhou; Seson Sun; Binyao Xu; Lingming Zhang; Hanshi Wang; Fudong Ge; Weiming Hu; Jin Gao

cs.ROOct 24, 2026

MOON: Learning When to Forget and How to Move On in Vision-and-Language Navigation

Authors: Jingmeng Zhou, Seson Sun, Binyao Xu, Lingming Zhang, Hanshi Wang, Fudong Ge, Weiming Hu, Jin Gao,

Organizations: AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University · State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA

Abstract

Recent vision-and-language navigation (VLN) policies act on their visual history, and an incorrect action can take them off the instructed route. Paired interventions on four VLN policies show that clearing the history doubles pooled success at the onset of a deviation but lowers it on route, while later in a deviation, removing only the off-route frames helps much less. After a deviation, the retained history keeps the policy from turning where a turn is needed, which we call behavioral inertia. We propose MOON, a framework that equips a frozen VLN policy with two lightweight modules, adding less than 1% of its parameters. A deviation judge decides when to forget, raising an alarm that clears the history when it predicts that the agent is off route. A recovery policy, trained with GRPO from alarm states, learns how to move on, turning more of the successes found only by sampling into a single greedy rollout. Experiments on R2R-CE and RxR-CE show that MOON reaches the highest R2R-CE success rate among the compared methods and raises the success rate of its frozen base by 5.2 and 6.0 points, and an ablation with a random trigger indicates that this gain depends on when the history is cleared. Without retraining, MOON also improves zero-shot object-goal navigation on HM3D-OVON. Together, these results suggest that deviation recovery depends on knowing when to forget, not only how to act. Code is available at https://github.com/AutoLab-SAI-SJTU/MOON.

Figures & tables

Figure 1: MOON learns when to forget and how to move on. Left: one paired episode in an unseen environment, in which the base fails and MOON recovers after the judge’s alarm (amber border), and the effect of clearing, which raises pooled success from 13.1% to 26.3% across 779 delayed states of four frozen policies. Middle: a deviation judge triggers history clearing and a trained recovery policy on a frozen 4B backbone. Right: R2R-CE and RxR-CE val-unseen results against recent leading monocular RGB models of Table 1 , 4B versions where a model comes in several sizes, and zero-shot HM3D-OVON results against JanusVLN and NavFoM. Each radar axis is scaled to the range of the plotted models; labeled values are MOON ’s.

Panel of Figure 2 — Figure 2: Visual history helps on route but can hinder recovery. (a) Boxed steps are cleared. (b, c) Off route: delayed states. Bands and whiskers: $95\%$ paired bootstrap intervals.

Panel of Figure 3 — Figure 3: Retained history keeps the policy from turning; after clearing, sampling finds more successes. (a) After a forward step, four policies pooled. (b) Onset states; oracle: the successful rollout. (c) Onset states whose retained greedy rollout failed. $95\%$ CIs: bootstrap (a, c), Wilson (b).

Figure 4: Overview of MOON. Top: at deployment, the deviation judge $f_{\phi}$ raises an alarm when the deviation score $\bar{m}_{t}$ exceeds the conformal threshold $\tau$ , clearing empties the visual history, and the recovery policy $\pi_{\theta,\psi}$ acts in one greedy rollout. Bottom: in training, $G$ rollouts are sampled from each alarm state and scored by the suffix-SPL reward, and their group-relative advantages update only the LoRA $\psi$ . Scenes and the score inset show one episode of the trained system, while the sampled rollouts are schematic and the advantage bars come from a group resampled after training.

Method

Params

Observation

R2R-CE Val-Unseen

RxR-CE Val-Unseen

Pano.

Odo.

Depth

S.RGB

\downarrow

\uparrow

\uparrow

SPL

\uparrow

\downarrow

\uparrow

SPL

\uparrow

nDTW

\uparrow

Panoramic or depth input

HPN+DN [ICCV21] [ 16 ]

–

✓

6.31

40.0

36.0

34.0

–

CMA [CVPR22] [ 12 ]

–

✓

6.20

52.0

41.0

36.0

8.76

26.6

22.2

47.1

ETPNav [TPAMI24] [ 1 ]

–

✓

4.71

65.0

57.0

49.0

5.64

54.8

44.9

61.9

NavMorph [ICCV25] [ 39 ]

–

✓

5.75

56.9

47.9

33.2

8.85

30.8

22.8

44.2

Table 1: Comparison with prior methods on the R2R-CE and RxR-CE val-unseen splits. Bold and underline mark the best and second-best results in each column. Pano., Odo., Depth and S.RGB denote panoramic view, odometry, depth and single RGB input. † Recovery-oriented method.

Table 7

Figure 7: Qualitative result on RxR-CE. Rows share frames until the alarm (amber border), after which MOON clears the history and the recovery policy acts; last column: end of each run.

Component	Params	Latency
Base (frozen)	4.44B	498 ms
+ Judge	+23.6M	–
+ Recovery	+12.4M	–
Full system	+36.0M $\uparrow$ 0.81%	507 ms $\uparrow$ 1.8%

Table 5: Parameters and latency

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Population	NaVIDA	StreamVLN	NaVILA	Uni-NaVid
On-route history controls	359	337	402	367
Delayed history interventions	205	210	163	201
Greedy onset interventions	238	248	198	231
Sampling: retained failures	80	80	80	80
Action continuation, onset and delayed	279	302	292	391

Appendix

Table 6: State counts for the analysis panels. After-deviation action observations include both onset and delayed states from some of the same episodes.

Construction	NaVIDA	StreamVLN	NaVILA	Uni-NaVid	Pooled
Prefix 0.5 m	$-7.3$ $[-14.2,0.0]$	$-14.8$ $[-22.4,-7.6]$	$-0.6$ $[-8.6,7.4]$	$-5.5$ $[-11.4,0.5]$	$-7.4$ $[-10.8,-3.7]$
Prefix 1.0 m	$-10.7$ $[-18.5,-3.4]$	$-12.4$ $[-19.5,-5.7]$	$-3.1$ $[-11.0,4.9]$	$-5.5$ $[-11.4,0.0]$	$-8.2$ $[-11.8,-4.6]$
Prefix 1.5 m	$-10.7$ $[-18.1,-3.4]$	$-12.4$ $[-19.5,-5.2]$	$-8.0$ $[-16.0,0.0]$	$-8.0$ $[-13.9,-2.5]$	$-9.9$ $[-13.2,-6.5]$
Prefix + current	$-9.3$ $[-16.6,-2.0]$	$-15.7$ $[-22.9,-9.0]$	$-4.3$ $[-12.3,3.7]$	$-9.5$ $[-15.4,-3.5]$	$-10.0$ $[-13.5,-6.5]$
Last 2	$-7.3$ $[-13.7,-1.0]$	$-3.8$ $[-8.6,1.0]$	$1.2$ $[-6.7,9.2]$	$-3.0$ $[-9.0,2.5]$	$-3.5$ $[-6.7,-0.3]$
Last 4	$-7.3$ $[-14.6,0.0]$	$-4.8$ $[-12.9,2.9]$	$0.6$ $[-7.4,8.6]$	$-4.0$ $[-9.5,1.5]$	$-4.1$ $[-7.8,-0.5]$

Appendix

Table 7: Success-rate differences from current-frame-only, in points; a negative value is a deficit of the same size in the main text. Each cell gives the estimate and its pointwise

95\%

paired state-bootstrap interval. The default population is

779

delayed states. Route-only uses

420

eligible states. Pooled estimates weight states equally.

On route
Policy	$n$	Retained	Cleared	Shuffled	Swapped	Expert
NaVIDA	172	39.0 [32.0, 45.9]	22.7 [16.3, 29.1]	39.5 [32.6, 47.1]	37.8 [30.8, 45.3]	32.0 [25.0, 38.4]
StreamVLN	219	36.5 [30.1, 42.9]	20.1 [15.1, 25.6]	39.7 [33.3, 46.1]	37.0 [30.6, 43.4]	32.4 [26.5, 38.8]
NaVILA	237	76.8 [71.3, 82.3]	18.1 [13.5, 23.2]	51.5 [45.6, 57.8]	68.1 [62.1, 74.0]	35.4 [29.1, 41.8]
Uni-NaVid	318	43.7 [38.1, 49.1]	21.1 [17.0, 25.8]	62.9 [57.2, 68.2]	56.0 [50.3, 61.4]	26.1 [21.4, 30.8]
Pooled	946	49.5 [46.2, 52.7]	20.4 [17.9, 22.8]	50.4 [47.3, 53.6]	51.3 [48.3, 54.6]	31.0 [28.1, 33.9]

Appendix

Table 8: Continuation rates per policy, in percent, with

95\%

episode-bootstrap intervals. States follow a forward step, and states where the expert returns no action are excluded. Pooled rows are the values in Figure 3(a) . Two on-route states each of NaVILA and Uni-NaVid have no swapped history, leaving

235

and

316

states (

942

pooled).

Policy	Retained $\mathrm{pass}@8$	Cleared $\mathrm{pass}@8$	Cleared: all fail
NaVIDA	36.2 [26.2, 47.5]	66.2 [56.2, 76.2]	33.8
StreamVLN	32.5 [22.5, 43.8]	68.8 [58.8, 78.8]	31.2
NaVILA	36.2 [26.2, 47.5]	71.2 [61.3, 80.0]	28.7
Uni-NaVid	16.2 [8.8, 25.0]	40.0 [30.0, 51.2]	60.0

Appendix

Table 9: Offline coverage after eight samples on retained-greedy failures (

80

onset states per policy). Values are percentages with

95\%

state-bootstrap intervals. The final column is the fraction with no successful cleared continuation among eight samples.

Configuration	Setting
Backbone	Qwen3-VL-4B-Instruct
Training data	$0.82$ M (R2R-CE and RxR-CE), $0.76$ M (DAgger) and $3.35$ M (ScaleVLN); $4.92$ M samples in total (components rounded)
Stages	Stage I on R2R-CE and RxR-CE, DAgger collection, Stage II on all data
Epochs	$1$ per stage
Learning rate	$2\times 10^{-5}$ , cosine decay
Global batch size	$64$ (Stage I), $128$ (Stage II)

Appendix

Table 10: Training configuration of the base policy NaVIDA-Qwen3VL.

Training	Start	Clear	SR $\uparrow$ at iteration
Training	Start	Clear	3	9	15	19
None (clear only)	–	–	65.6
SFT	alarm	✓	61.8	62.9	–	–
GRPO ${}_{\mathrm{start}}^{*}$	step 0	–	–	62.3	–	–
GRPO start	step 0	–	–	66.2	–	–
GRPO keep	alarm	✗	65.5	66.2	67.4	67.8

Appendix

Table 11: Training-side ablation on R2R-CE: SR at iterations 3, 9, 15 and 19, counted from 0. Start: where rollouts or demonstrations begin. Clear: whether they begin from cleared history; the row without it also retains the history at deployment. ∗ No judge at deployment. GRPO binary : binary success reward instead of suffix-SPL. –: does not apply or was not run.

	R2R-CE ( $\alpha=10\%$ )	RxR-CE ( $\alpha=7.5\%$ )
Alarm	527/1,839	779/3,669
Prec.	375/527	589/779
Recall	375/666	589/1,286
False alarm	152/1,173	190/2,383
Rescued	164/375 [38.8, 48.8]	317/589 [49.8, 57.8]
Broken	68/152 [37.1, 52.7]	97/190 [44.0, 58.1]

Appendix

Table 12: Counts behind Figure 5 . Brackets: Wilson

95\%

intervals. RxR-CE recall includes 99 base failures whose goal lies on a disconnected part of the navigation mesh, and is 589/1,187 without them.

Dataset	$\alpha$	$\tau$	Alarms	False alarms	SR $\uparrow$	SPL $\uparrow$
R2R-CE	5.0%	3.83	16.3	5.5	66.9	60.3
	7.5%	3.08	23.3	9.4	67.9	60.4
	10.0% †	2.54	28.7	13.0	69.0	62.3
	12.5%	2.29	30.7	14.7	67.8	59.7
	Base	–	–	–	63.8	59.0
RxR-CE	2.5%	4.37	11.2	3.1	67.0	56.7

Appendix

Table 13: Alarm level. Alarms: share of all episodes; false alarms: share of base successes. † Reported level.

Interval (decision points)	SR $\uparrow$	SPL $\uparrow$	Alarms
1 †	69.0	62.3	527
2	67.9	59.9	535
4	67.0	58.8	540

Appendix

Table 14: Judge query interval on R2R-CE. † Deployed interval.

Method	SR $\uparrow$	SPL $\uparrow$
DAgRL+OD [IROS24] [ 42 ]	37.1	19.9
MTU3D † [ICCV25] [ 53 ]	40.8	12.1
ABot-N0 [arXiv26] [ 7 ]	54.0	30.5
Qwen-RobotNav-4B [arXiv26] [ 50 ]	53.1	20.9
Qwen-RobotNav-8B [arXiv26] [ 50 ]	51.2	24.0
VLFM ∗ [ICRA24] [ 41 ]	35.2	19.6

Appendix

Table 15: Comparison on the HM3D-OVON val-unseen split. Results of other methods are taken from their papers, except VLFM, whose results are from Yokoyama et al. [42] . Bold and underline mark the best and second-best results. ∗ Zero-shot evaluation, without training on HM3D-OVON. VLFM, DAgRL+OD and MTU3D also use depth and odometry, ABot-N0 uses panoramic RGB and NavFoM four cameras. † 360 randomly sampled episodes.

Figure 8: Additional R2R-CE episodes (1 of 3) in which the frozen policy fails and MOON succeeds, shown as in Figure 7 .

Figure 9: Additional R2R-CE episodes (2 of 3), shown as in Figure 7 .

Figure 10: Additional R2R-CE episodes (3 of 3), shown as in Figure 7 .

Figure 11: Additional RxR-CE episodes (1 of 3) in which the frozen policy fails and MOON succeeds, shown as in Figure 7 .

Figure 12: Additional RxR-CE episodes (2 of 3), shown as in Figure 7 .

Figure 13: Additional RxR-CE episodes (3 of 3), shown as in Figure 7 .

Vision-Language Navigation

Vision-Language Navigation studies how embodied agents use natural-language instructions and visual observations to plan and execute routes.

See all 313 papers

ReInsVLN: Revisiting Historical Evidence for Training-Free Vision-and-Language Navigation
Oct 19, 2026Siwei Cheng, Hao Wu, Zitong Qi +7Vision-Language Navigation Memory-Augmented VLMs
University of Science and Technology of China · Eastern Institute of Technology, Ningbo · Shanghai Jiao Tong University
AVERT-VLN: Abstention-aware Visual Error Recovery and Training for Vision-and-Language Navigation
Sep 30, 2026Minrui Liu, Jingke Wang, Yuehao Huang +4Vision-Language Navigation
Institute of Cyber-Systems and Control, Zhejiang University, Hangzhou, China.
What Should a Navigator Remember? Spatiotemporal Memory for Zero-Shot Vision-Language Navigation
Oct 28, 2026Yuhang Peng, Zhikai Wang, Ziyi Wang +5Vision-Language Navigation Memory-Augmented VLMs
The Hong Kong Polytechnic University · Tsinghua University · Shanghai Jiao Tong University

Oct 19, 2026J/K move · Enter open · S save

ReInsVLN: Revisiting Historical Evidence for Training-Free Vision-and-Language Navigation

Siwei Cheng, Hao Wu, Zitong Qi, Xinming Zhang+6

University of Science and Technology of China · Eastern Institute of Technology, Ningbo · Shanghai Jiao Tong University

Vision-Language Navigation Memory-Augmented VLMs VLN Benchmarks

Open paper AskPDF arXiv

Vision-and-language navigation (VLN) relies on retaining and using relevant observations. Yet growing histories introduce redundancy, and retained information may remain underused. We introduce ReInsBench, a diagnostic benchmark that examines history utilization through controlled continuation trials. Its results show that history-review reinstruction can recover navigation failures using the existing task context. Motivated by this finding, we propose ReInsVLN, a training-free framework supporting memory construction and utilization in frozen navigators. Coverage- and Instruction-Aware Compression constructs compact visual context through temporal redundancy reduction, coverage-aware protection, and instruction-guided token selection. Forgetting-Inspired History Reminders guide navigators to revisit retained observations according to a predefined schedule, without additional model calls. On R2R Val-Unseen, ReInsVLN improves success rates by 8.48 and 3.37 percentage points over the native StreamVLN and InternVLA-N1, reaching 64.27% and 64.44%, respectively. Visual compression largely preserves navigation performance while reducing prefill latency and KV cache memory by up to 21.77% and 32.25%, respectively, on matched observation workloads. Real-world trials on humanoid and quadruped robots further support the applicability of ReInsVLN to sequential instruction following and ordinal navigation.

Abstract

Vision-Language Navigation

Explore similar work

ReInsVLN: Revisiting Historical Evidence for Training-Free Vision-and-Language Navigation

AVERT-VLN: Abstention-aware Visual Error Recovery and Training for Vision-and-Language Navigation

What Should a Navigator Remember? Spatiotemporal Memory for Zero-Shot Vision-Language Navigation