FINE: Future-Informed Navigation Encoding for Data-Efficient Vision-Language Navigation

Khang H. Nguyen; Hoang Pham Quang Nguyen; Ha Phuong Nguyen; Khanh Dinh Binh; Xuan Ha Nguyen; Vien Ngo; Duy Ho Nguyen Minh; Huan Nguyen

cs.ROSep 26, 2026

FINE: Future-Informed Navigation Encoding for Data-Efficient Vision-Language Navigation

Authors: Khang H. Nguyen, Hoang Pham Quang Nguyen, Ha Phuong Nguyen, Khanh Dinh Binh, Xuan Ha Nguyen, Vien Ngo, Duy Ho Nguyen Minh, Huan Nguyen,

Organizations: VinRobotics · University of California, Los Angeles (UCLA) · Center for AI Research, VinUniversity, Vietnam · German Research Center for Artificial Intelligence (DFKI) ·

Abstract

Adapting vision-language navigation (VLN) policies to new environments is expensive because every additional route and instruction requires an embodied demonstration. Yet standard observation-to-action training uses only a small fraction of the information already contained in each trajectory. In particular, future observations reveal the instruction-relevant landmarks that the agent will encounter, including what they look like and how they are arranged in 3D. We introduce FINE, a Future-Informed Navigation Encoding framework that extracts this latent supervision from existing demonstrations. FINE equips a VLN backbone with two complementary auxiliary representations. First, explicit landmark tokens follow the ordered landmarks specified by the instruction and are trained to predict future landmark regions in both semantic 2D patch-feature space and viewpoint-dependent 3D geometric feature space. Second, an implicit future token learns to distinguish the landmark state that is actually reached from plausible same-scene counterfactual futures generated by a video world model. On R2R-CE and RxR-CE val-unseen, FINE improves InternVLA-N1 by 2.6 and 4.5 success-rate points, respectively, at full training data. More importantly, as demonstrations become limited, the benefit grows: at a 70% demonstration budget, FINE improves success rate by 6.8 points, recovering roughly one-third of the performance lost by reducing the training demonstrations. Project page is available at https://finevln.github.io/.

Figures & tables

Fig. 1: A VLN demonstration provides supervision beyond action through future landmark observations. Top : a realworld rollout following carton box $\rightarrow$ blue table $\rightarrow$ orange chair $\rightarrow$ green plant; observations 2 and 5 contain no relevant landmark. Bottom-left : explicit future learning predicts semantic and 3D geometric features of the upcoming landmark. Bottom-right : implicit future learning distinguishes the realized future from same-scene counterfactuals.

Fig. 2: Bottom (train and inference): given the instruction, RGB history, and current observation at time $t$ , the object-aware encoder constructs ordered explicit landmark tokens $\mathbf{E}^{\mathrm{exp}}_{t}=\{\mathbf{e}^{\mathrm{exp}}_{t,j}\}_{j=1}^{S}$ and an implicit future token $\mathbf{e}^{\mathrm{imp}}_{t}$ . After VLM processing, the resulting representations $\widetilde{\mathbf{E}}^{\mathrm{exp}}_{t}$ and $\tilde{\mathbf{e}}^{\mathrm{imp}}_{t}$ , together with the shared context $\mathbf{h}_{t}$ , condition the original action decoder. Top (training only): the explicit branch decodes each landmark token into semantic predictions $\hat{\mathbf{s}}_{j,p}$ and 3D geometric predictions $\hat{\mathbf{g}}_{j,p}$ for the corresponding future landmark region, matched to frozen targets $\mathbf{s}_{j,p}$ and $\mathbf{g}_{j,p}$ . The implicit branch projects $\tilde{\mathbf{e}}^{\mathrm{imp}}_{t}$ to $\mathbf{z}_{t}$ , which is pulled toward the realized future-landmark representation $\mathbf{y}_{t^{+}}$ and pushed away from cross-episode and same-scene counterfactual futures.

Budget	Method	NE $\downarrow$	OS $\uparrow$	SR $\uparrow$	SPL $\uparrow$
100%	NaVILA [ 3 ]	5.20	61.6	54.0	49.0
	$+$ FINE	5.00	63.1	55.6	50.7
	InternVLA-N1 [ 5 ]	4.83	63.3	58.2	54.0
	$+$ FINE	4.29	65.7	60.8	55.2
70%	NaVILA [ 3 ]	6.76	46.9	36.2	29.5
	$+$ FINE	6.19	50.6	41.7	35.5

TABLE I: Demonstration-budget comparison on R2R-CE val-unseen. Baseline and FINE use identical training subsets and matched optimization within each budget. Differences are in points.

Method	Observation				R2R-CE Val-Unseen				RxR-CE Val-Unseen
Method	Pano.	Odo.	Depth	S.RGB	NE $\downarrow$	OS $\uparrow$	SR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	SR $\uparrow$	SPL $\uparrow$	nDTW $\uparrow$
CMA [ 32 ]	✓	✓	✓		6.20	52.0	41.0	36.0	8.76	26.5	22.1	47.0
ETPNav [ 15 ]	✓	✓	✓		4.71	65.0	57.0	49.0	5.64	54.7	44.8	61.9
ScaleVLN [ 4 ]	✓	✓	✓		4.80	–	55.0	51.0	–	–	–	–
NaVid [ 1 ]				✓	5.47	49.1	37.4	35.9	–	–	–	–
UniNaVid [ 6 ]				✓	5.58	53.3	47.0	42.7	6.24	48.7	40.9	–

TABLE II: VLN-CE R2R and RxR val-unseen results at 100% training data. Prior results are from original publications and may differ in observation space and external data; InternVLA-N1 DualVLN and NaVILA are evaluated from the authors’ released checkpoint. Ours uses no external data 2 2 2 Refers to VLN demonstration data; see Sec. IV-C and Table V for the auxiliary world model’s role. ; JanusVLN

{\dagger}

uses

\sim

10.7M external image-action pairs. Pano.: panoramic RGB; Odo.: odometry; S.RGB: single-view RGB. Best in bold , second best underlined ; –: not reported.

Model	Data	Experiment 1	Experiment 2	Experiment 3
NaVILA	100%	60 / 2.3	10 / 8.8	0 / 10
$+$ FINE	100%	65 / 2.2	15 / 7.2	10 / 8.5
InternVLA-N1	100%	80 / 0.4	30 / 3.5	30 / 5.5
$+$ FINE	100%	85 / 0.4	60 / 2.3	50 / 4.2
InternVLA-N1	70%	60 / 1.7	10 / 6.4	10 / 6.7
$+$ FINE	70%	75 / 1.5	25 / 3.7	25 / 4.2

TABLE III: Real-world evaluation with SR (%)

\uparrow

and NE (m)

\downarrow

. Experiments 1–2: single-room (visible and occluded landmarks); Experiment 3: room-to-room (multiple landmarks).

	100%				70%
Method	NE $\downarrow$	OS $\uparrow$	SR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	OS $\uparrow$	SR $\uparrow$	SPL $\uparrow$
Baseline	4.83	63.3	58.2	54.0	6.45	46.0	38.5	34.4
$+$ explicit only	4.38	64.4	59.5	53.3	5.96	53.8	44.4	38.3
$+$ implicit only	4.46	65.7	59.6	54.4	5.58	48.6	42.5	37.4
$+$ FINE (full)	4.29	65.7	60.8	55.2	5.58	56.0	45.3	40.2

TABLE IV: Contribution of the two FINE branches on R2R-CE val-unseen, at 100% and 70% demonstrations.

Base	Variant	NE $\downarrow$	OS $\uparrow$	SR $\uparrow$	SPL $\uparrow$
	$S=2$ slots	6.21	47.0	40.0	36.0
Expl.	$S=10$ slots	5.96	53.8	44.4	38.3
	$S=18$ slots	5.96	49.0	41.3	36.3
Expl.	Semantic only	6.04	48.8	42.0	37.2
	Geometric only	5.96	50.0	42.9	37.7
	Semantic $+$ Geometric	5.96	53.8	44.4	38.3

TABLE V: Design choices on R2R-CE val-unseen at the 70% budget (Table IV baseline: 38.5 SR). Base marks rows built on the Expl. (explicit-only) or Impl. / Full configuration; see text for details.

Fig. 3: Qualitative results from the real-world deployment of FINE. Given an instruction, the robot moves through different areas of the workspace and stops at the specified goal.

Vision-Language Navigation

Vision-Language Navigation studies how embodied agents use natural-language instructions and visual observations to plan and execute routes.

See all 296 papers

Anticipate Before Acting: Future-State-Conditioned Vision-Language Navigation
Jul 20, 2026Lingfeng Zhang, Zhanguang Zhang, Liheng Ma +2Visual Navigation Vision-Language Navigation
Noah’s Ark Lab, 2012 Labs, Huawei · Department of Foundation model, 2012 Labs, Huawei
FutureNav: Unified World-Action Modeling for Vision-and-Language Navigation
Jun 29, 2026Lingfeng Zhang, Zeying Gong, Xiaoshuai Hao +7World Models Vision-Language Navigation
Tsinghua University · Pengcheng Laboratory · Xiaomi EV +2
GuidedNav: Shaping Vision-Language Navigation Representations through Pre-action Attention and Spatial Guidance
Oct 9, 2026Jingyu Guo, Jiaxin Huang, Chengxing Lin +6Vision-Language Navigation Visual Navigation
University of Melbourne · AgiBot · Mohamed bin Zayed University of Artificial Intelligence +3

Jul 20, 2026J/K move · Enter open · S save

Anticipate Before Acting: Future-State-Conditioned Vision-Language Navigation

Lingfeng Zhang, Zhanguang Zhang, Liheng Ma, Tongtong Cao+1

Noah’s Ark Lab, 2012 Labs, Huawei · Department of Foundation model, 2012 Labs, Huawei

Visual Navigation Vision-Language Navigation Predictive Representation Learning

Open paper AskPDF arXiv

End-to-end vision-language navigation (VLN) with causal vision-language models maps instructions and egocentric observations directly to actions, but standard behavior cloning supervises only the next action and does not explicitly encourage the policy state to be predictive of future visual outcomes, limiting long-horizon decision making. A privileged-input diagnostic shows that access to an expert-trajectory future image can substantially improve navigation, indicating that future observations contain rich, actionable cues, though such inputs are unavailable at deployment. Motivated by this signal, we propose Future-State-Conditioned VLN (FSC-VLN), a deployable model that augments a causal policy with a future-query token and uses training-only future-state supervision to distill information from future observations into the policy state. Concretely, during training we align the future-query representation to a frozen visual embedding

Δ

steps ahead, while inference requires only past and current observations. This design preserves the baseline inference pattern and adds only two learned prefix tokens, implying minimal overhead. On R2R val-unseen, FSC-VLN improves SR/OSR/SPL over a StreamVLN-style baseline under two training-data regimes, with larger gains on long-horizon episodes; ablations further support the dual-query design that separates future and action queries.

Abstract

Vision-Language Navigation

Explore similar work

Anticipate Before Acting: Future-State-Conditioned Vision-Language Navigation

FutureNav: Unified World-Action Modeling for Vision-and-Language Navigation

GuidedNav: Shaping Vision-Language Navigation Representations through Pre-action Attention and Spatial Guidance