LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigation

Bo Miao; Weijia Liu; Jun Luo; Lachlan Shinnick; Jian Liu; Thomas Hamilton-Smith; Yuhe Yang; Zijie Wu

cs.CVFeb 2, 2026

LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigation

Authors: Bo Miao, Weijia Liu, Jun Luo, Lachlan Shinnick, Jian Liu, Thomas Hamilton-Smith, Yuhe Yang, Zijie Wu, +3 more

Organizations: AIML, Adelaide University · East China Normal University

Abstract

Language-conditioned goal navigation (LGN) requires embodied agents to locate user-specified targets without step-by-step guidance. However, existing benchmarks largely focus on category-level goals or rely on instance descriptions generated by vision-language models, which often contain ambiguities and semantic errors, limiting systematic and reliable evaluation. We introduce HieraNav, an open-vocabulary LGN task with goals at four hierarchical semantic levels: scene, room, region, and instance. We present Language as a Map (LangMap), the first LGN benchmark to enrich real-world indoor 3D scans with human-verified semantic annotations supporting tasks across all four goal levels. Built on HM3D using a contrastive annotation protocol that compares same-scene regions and instances, LangMap provides region labels and discriminative region and instance descriptions covering 414 object categories and contains over 18K tasks. Each target has concise and detailed descriptions, enabling evaluation across instruction styles. Automated and human evaluations validate our annotation quality: our descriptions improve text-to-view matching accuracy over GOAT-Bench's by 23 points on all shared annotated instances, and an independent human audit yields 92.5% unique-and-correct matches. We also propose PlaNaVid, an RGB-only baseline that combines Bounded Diverse Memory with high-level planning to prime a reactive policy for multi-goal navigation, achieving top-tier success rates without depth, 3D scene representations, or object masks. Further analyses reveal that exploration and hierarchical disambiguation failures become more prominent at finer goal levels, while long-tail categories, small objects, distant targets, timely stopping, and multi-goal completion remain challenging. Benchmark and code: https://bo-miao.github.io/LangMap

Figures & tables

Figure 3 : HieraNav requires agents to navigate to language-specified targets at four levels: scene, room, region, and instance. LangMap enables evaluation with mixed-level multi-goal episodes and single-goal tasks. Descriptions cover intrinsic attributes, spatial context, and open-world knowledge.

Eval Benchmark

Goal Granularity

Region Annotation

Object Annotation

Small Obj

Tasks

Scene

Room

Region

Instance

Cat

Desc

Words

Cat

Desc

Words

RoboTHOR [ 65 ]

✓

\times

\times

\times

\times

\times

ObjNav-MP3D [ 21 ]

✓

\times

\times

\times

\times

\times

ObjNav-HM3D [ 2 ]

✓

\times

\times

\times

\times

\times

HM3D-OVON [ 1 ]

✓

\times

\times

\times

178

\times

\times

4.2%

9000

LHPR-VLN [ 6 ]

✓ †

\times

\times

\times

\times

960

Table 1: Statistics of LGN evaluation benchmarks. Cat/Desc/Words: number of categories, number of discriminative descriptions, and average word count (concise/detailed). Small Obj: percentage of objects with mean frame coverage below 3.3% across look-down, forward, and look-up views. Blanks and

\times

indicate unsupported settings. † : VLM-generated descriptions (GOAT-Bench) and inferred room types (LHPR-VLN). LangMap provides human-verified multi-granularity annotations.

Figure 4 : LangMap distributions. (a) Region label frequency. (b) Instances per object category, with sampled labels. (c) Ground-truth geodesic distances of navigation tasks across four semantic levels.

Figure 5 : Overview of our data processing pipeline.

Figure 6 : Examples of the contrastive annotation process. (a) Region annotation: annotators compare same-category regions within a scene. (b) Instance annotation: annotators compare same-category or semantically related instances using object views and verified region context. All annotations are cross-checked for correctness and discriminability.

Figure 7 : Overview of PlaNaVid, a decoupled RGB-only framework using RGB and language without depth, 3D representations, or object masks. Bounded Diverse Memory maintains a bounded history with near-uniform temporal coverage and provides diverse context for a VLM planner, which selects an initial waypoint and heading per goal to prime reactive policy for multi-goal navigation.

Method	Observation	Mask	3D Map	Local Nav.	Multi-Goal			Single-Goal		Latency (s/goal)
Method	Observation	Mask	3D Map	Local Nav.	SR $\uparrow$	SeqSR@2 $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	Latency (s/goal)
3D-Mem-3B [ 68 ]	RGB-D (pano.)	✓	✓	Shortest	13.5	1.1	6.4	8.5	1.9	147.2
3D-Mem-7B [ 68 ]	RGB-D (pano.)	✓	✓	Shortest	30.1	5.1	17.3	13.7	6.2	95.6
3D-Mem-3B † [ 68 ]	RGB-D (pano.)	✓	✓	Shortest	20.4	2.2	11.3	15.3	2.8	147.2
3D-Mem-7B † [ 68 ]	RGB-D (pano.)	✓	✓	Shortest	36.8	10.0	21.2	21.2	8.7	95.6
MTU3D [ 69 ]	RGB-D (pano.)	✓	✓	Shortest	41.2	11.0	24.3	29.9	15.3	52.1

Table 2: Evaluation on LangMap with concise descriptions . -3B/-7B denote planner sizes. † Reference only: 3D-Mem’s original success criterion (within 1m of a target viewpoint, allowing up to 2m from the goal). Local Nav.: simulator pathfinder (Shortest) or learned policy (Learned).

Config	Method	Scene		Room		Region		Region-D		Instance		Instance-D
		SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$
RGB-D,Mask	3D-Mem-3B	11.2	2.5	11.8	2.8	8.2	1.8	6.8	1.4	4.7	1.1	5.1	1.1
	3D-Mem-7B	16.4	7.2	12.6	6.0	14.0	6.1	15.1	5.6	13.0	5.8	16.3	7.0
	MTU3D	31.4	15.7	32.5	16.2	33.6	16.8	33.7	16.2	23.8	12.9	28.7	15.2
RGB	PSL	6.0	1.4	6.6	1.9	7.3	2.1	9.0	2.6	6.4	1.9	8.5	2.4
	SenseAct-M	10.2	5.6	8.3	4.4	8.5	4.3	9.5	4.8	7.7	4.0	7.4	4.1

Table 3: Single-goal results on LangMap. Descriptions are concise unless marked ‘-D’ (detailed).

Table 10

Method	Category Frequency				Path Length						Object Size
Method	Head		Long-tail		Short		Medium		Long		Non-small		Small
	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$
3D-Mem-7B	14.4	6.6	11.1	4.6	27.3	12.8	11.6	4.9	4.4	2.0	14.7	6.7	11.0	4.6
MTU3D	32.1	16.3	22.0	11.5	46.3	21.7	28.0	14.8	17.4	9.7	32.8	16.7	20.5	10.0
Uni-NaVid	31.3	15.9	26.1	13.2	42.2	21.5	29.7	14.7	19.4	10.3	32.6	16.5	23.0	10.9
PlaNaVid-3B	32.4	15.8	27.4	13.0	43.8	21.7	30.5	14.4	20.7	10.4	34.0	16.4	22.4	10.5

Table 6: Single-goal analysis by category frequency, optimal path length, and object size.

Components			Results
Mem	GUU	SDR	SR	SeqSR	SPL	#Frames $\downarrow$
			34.4	10.4	15.0	-
✓			40.4	13.2	16.9	656
✓	✓		40.9	13.2	17.3	50
✓		✓	42.0	14.2	17.8	629
✓	✓	✓	42.8	14.6	18.1	50

Table 7 : Ablation of PlaNaVid.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8 : Representative object-view and region-view capture.

Figure 9 : Comparison of instance-level descriptions in GOAT-Bench and LangMap. Blue marks semantic errors and green marks ambiguous descriptions that match multiple objects. Our human-verified descriptions provide correct, fine-grained semantics and uniquely refer to the target instance in the scene.

Figure 10 : Our annotation comparison viewer for qualitative analysis. It displays the target and same-category distractors to assess whether each description uniquely identifies the target.

Figure 11 : Failure cases of PlaNaVid. Green boxes mark ground-truth targets and red boxes mark distractors at the stop location.

Method	SeqSR@1	SeqSR@2	SeqSR@3	SeqSR@4	SeqSR@5
PSL	0.0	0.0	0.0	0.0	0.0
SenseAct-M	6.0	1.0	0.1	0.1	0.1
3D-Mem-3B	6.9	1.1	0.3	0.0	0.0
3D-Mem-7B	12.4	5.1	1.9	0.8	0.1
MTU3D	25.0	11.0	5.4	2.5	1.1
Uni-NaVid	27.1	10.4	3.5	1.4	0.6

Appendix

Table 8: SeqSR@1–5 Results on LangMap.

	$N_{\max}$	$K$	SR	SeqSR@2	SPL
Latest Memory	50	10	36.4	11.4	14.9
Full Memory	656	10	40.4	13.2	16.9
Ours	25	10	41.5	13.8	17.3
	50	5	40.0	12.2	17.0
	50	10	42.8	14.6	18.1
	75	10	42.3	13.8	17.7

Appendix

Table 9: Ablation of

N_{\max}

and

K

. Ours uses Bounded Diverse Memory.

Config	Method	SR $\uparrow$	SeqSR@2 $\uparrow$	SPL $\uparrow$
RGB-D,Mask	3D-Mem-3B	13.5 [11.8, 15.2]	1.1 [0.4, 1.9]	6.4 [4.9, 8.1]
	3D-Mem-7B	30.1 [26.6, 33.7]	5.1 [3.1, 7.4]	17.3 [15.0, 19.8]
	MTU3D	41.2 [37.8, 44.7]	11.0 [8.1, 14.0]	24.3 [21.9, 26.7]
	MTU3D ∗	38.0 [34.5, 41.4]	7.9 [5.4, 10.6]	21.0 [18.8, 23.1]
RGB	Uni-NaVid	34.4 [31.0, 37.6]	10.4 [7.9, 12.9]	15.0 [13.3, 16.7]
	PlaNaVid-3B	41.5 [37.9, 44.9]	14.4 [11.1, 17.9]	17.5 [15.5, 19.5]

Appendix

Table 10: Multi-goal results (%) with 95% scene-cluster bootstrap CIs. ∗ : MTU3D with a horizontal field of view matched to PlaNaVid.

Figure 12 : Example of sequential multi-goal navigation. The colored paths denote different goals.

Explore similar work

Jul 4, 2026cs.RO

LH-AVLN: A Benchmark for Long-Horizon Audio-Visual-Language Navigation

Embodied navigation is moving toward long-horizon missions, yet existing long-horizon benchmarks are largely acoustically silent, and audio-visual navigation tasks typically focus on a single goal. We introduce LH-AVLN, a benchmark for Long-Horizon Audio-Visual-Language Navigation that combines multi-goal mission execution, heterogeneous goal specifications, and persistent spatialized acoustic cues. In LH-AVLN, an agent receives a global mission of two to four goals specified by category, language description, or reference image, and navigates with RGB-D observations, pose, and binaural audio in indoor 3D environments. The benchmark supports both ordered and unordered missions, where alternating goal-associated sounds can guide non-line-of-sight search but may also become distractors as mission progress changes. We further develop PAG-Nav, a training-free reference agent that maintains a temporal uniform semantic map and performs progressive goal-state planning, using sound for search while reserving completion for visual-semantic verification. Experiments show that existing vision-language, memory-based, and audio-visual agents struggle to complete full LH-AVLN missions, and that PAG-Nav provides a stronger diagnostic baseline while leaving substantial room for future progress.

Rufeng Chen, Yue Chang, Zili Shao +4

The Hong Kong University of Science and Technology (Guangzhou) · Jilin University

Vision-Language Navigation

Ask PDF

Jul 14, 2026cs.RO

Instance-Enriched Semantic Maps for Visual Language Navigation

Visual Language Navigation (VLN) aims to enable an embodied agent to navigate complex environments by following natural language instructions. Recent approaches build semantic spatial maps and leverage Large Language Models (LLMs) for reasoning and decision making. Despite these advances, existing systems lack instance-level object detail and robustness to diverse user queries, limiting reliable navigation in complex indoor environments. To address these limitations, we propose Instance-Enriched Semantic Maps, a unified framework with three key contributions: (1) Instance-level two-and-a-half-dimensional (2.5D) rich information mapping that constructs maps from color and depth observations via open-vocabulary panoptic segmentation, preserving vertical distinctions and capturing small objects, while storing diverse semantic attributes and natural language captions enriched with room-level context. (2) Robust query processing via LLM-based target selection, which dynamically routes queries across type-specialized experts and integrates their outputs through score-level fusion, enabling consistent goal selection across diverse query formulations. (3) Storage-efficient semantic representation that achieves approximately 96% reduction compared to three-dimensional (3D) scene-graph approaches while preserving sufficient spatial information for navigation. The proposed 2.5D representation outperforms the 3D baseline by over 27% in prediction-normalized Area Under the Curve (AUC). In navigation experiments, our method achieves over 17% improvement in object retrieval and over 23% in navigation success compared to the baseline across diverse query types. The project page is available at https://rcilab.github.io/iesm_vln.

Jiho Hong, Eunae Kang, Sanghyun Kim +1

Department of Mechanical Engineering, Kyung Hee University, Yongin-si, 17104, Republic of Korea · Advanced Institutes of Convergence Technology (AICT), Suwon, 16229, Republic of Korea · School of Mechanical Engineering, Kyungpook National University, Daegu, 41566, Republic of Korea

Vision-Language Navigation Semantic Mapping Object Goal Navigation Navigation Panoptic Segmentation

Ask PDF

Sep 14, 2026cs.RO

AnchorVLN: Geometry-Anchored Vision-Language Grounding Reasoning for Open-Vocabulary Navigation

Vision-Language Navigation (VLN) in unseen indoor environments is useful in real-world robotics, where an agent must follow natural-language instructions, locate objects, and answer spatial questions without a pre-built map or fixed object vocabulary. Multimodal vision-language models (VLMs) provide strong open-vocabulary grounding and zero-shot reasoning, but struggle to emit reliable metric quantities such as range, bearing, and comparative spatial relations directly from images. Existing approaches address this by folding geometry into hand-engineered pipelines or asking models to output waypoints, requiring changes to the control stack for different robots, tasks, or vocabularies. We introduce AnchorVLN, an open-vocabulary VLN system built on a simple rule: the VLM proposes semantics; geometry decides metrics. It is realised as EMBODIED-NAV-MCP, a Model Context Protocol (MCP) server driven by a VLM agent through a compact set of callable tools. Since no tool accepts distance in metres or bearing in radians, the schema enforces the semantic-geometry boundary without modifying the downstream autonomy stack. We benchmark both tasks of the CMU Vision-Language Navigation Challenge 2026: 30 instruction-following questions over 15 scenes and a frozen 45-question object-reference set. The full system achieves 64.4 percent on instruction following, dropping by 13.3 percentage points without controller modeling (t = 2.77). On object reference, geometric anchoring clears the challenge overlap threshold on 10 of 45 questions, versus 0 of 45 for direct coordinate estimation, reducing median center error from 3.37 m to 2.48 m.

Long Giang Vu, Chengkai Yao, Yuxin Liu +2

*These authors contributed equally.

Vision-Language Navigation Navigation

Ask PDF