HGWM: Hierarchical Graph-Guided World Model for Zero-Shot Object Navigation via Graph Matching

I-Tak Ieong; Hao Tang

cs.ROOct 23, 2026

HGWM: Hierarchical Graph-Guided World Model for Zero-Shot Object Navigation via Graph Matching

Authors: I-Tak Ieong, Hao Tang

Organizations: Tsinghua University · Peking University

Abstract

Zero-shot object navigation requires an embodied agent to choose where to search while retaining evidence from previous observations. HGWM couples an LLM-generated room-anchor prior with persistent spatial-semantic memory. A coarse stage assigns visual-semantic scores to candidate headings on a 0-10 scale; a fine stage measures symbolic compatibility between observed objects, room context, and the goal prior. Phase-dependent fusion combines these scores with an LLM direction recommendation before local action selection. On HM3D v0.1, the reported HGWM configuration achieves 62.6% success rate (SR) and 31.5% success weighted by path length (SPL), compared with 58.1% and 31.2% for WMNav. On MP3D, the corresponding values are 46.8% and 17.9%, compared with 45.4% and 17.2%. The results indicate higher success point estimates and modest path-efficiency gains. On HM3D v0.2, the full model obtains 74.4% SR and 31.9% SPL. Additional component ablations reduce SR by 0.9-2.3 percentage points and SPL by 0.6-1.5 points. These are author-reported aggregate measurements. The accompanying implementation repairs pass offline tests; episode-level rerun records are not available. Evaluation is confined to simulation.

Figures & tables

Figure 1: HGWM architecture. The LLM generates a hypothetical room–anchor prior, observations populate spatial-semantic memory, and graph-guided reasoning informs navigation. Hallucinated Prior and Structural Verification denote prior generation and heuristic compatibility checks; neither establishes scene truth. System 1/2 labels are conceptual analogies.

Figure 2: Coarse-to-fine illustration with example trajectory views. Coarse VLM scores use 0–10: 0 for a dead end without a visible target, 1–9 for intermediate opportunities, and 10 for a model-reported visible target. The example fine scores are heuristic compatibility values, not calibrated probabilities. The ordered phase thresholds are defined in the text.

Figure 3: Fine-matching illustration. Corrected Scene Graph denotes the graph after heuristic relation proposals, not verified truth. The illustrated embedding similarity is interpreted as symbolic feature comparison, without a learned graph-embedding network. Equation 6 specifies all five features, including semantic and depth terms in addition to the illustrated object, room, and spatial terms.

Phase	$a$ (coarse)	$b$ (fine)	$e$ (LLM)
Frontier exploration	0.6	0.1	0.5
Focused search	0.6	0.4	0.4
Target verification	0.8	0.5	0.2

Table 1: Phase-dependent fusion coefficients.

Figure 4: Decision process comparison between WMNav and HGWM. In these selected examples, HGWM visits different regions and exhibits less repeated exploration. These trajectories illustrate behavior; they do not establish an aggregate reduction in looping or isolate the effect of graph matching.

Method	Vision	Language	HM3D v0.1		MP3D
Method	Vision	Language	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$
Habitat-Web [ 19 ]	-	-	41.5	16.0	31.6	8.5
OVRL [ 24 ]	-	-	-	-	28.6	7.4
OVRL-V2 [ 23 ]	-	-	64.7	28.1	-	-
ZSON [ 16 ]	CLIP	-	25.5	12.6	15.3	4.8
PSL [ 21 ]	CLIP	-	42.4	19.2	18.9	6.4

Table 2: Comparison on HM3D v0.1 and MP3D. SR and SPL are percentages; higher is better. Baseline results are taken from the cited papers and use their respective training and model settings. For HGWM on HM3D v0.1, superscripts and subscripts give the maximum and minimum deviations across three runs, not confidence intervals.

Comparison	$\Delta$ SR	$\Delta$ SPL
HGWM–WMNav, HM3D v0.1	+4.5	+0.3
HGWM–WMNav, MP3D	+1.4	+0.7
Voxel–no memory, HM3D v0.2	+3.1	+0.8
HGWM–voxel, HM3D v0.2	+3.5	+1.8

Table 3: Differences between configurations in percentage points, computed from the corresponding SR and SPL values.

VLM Backbone	SR (%)	SPL (%)
Gemini-1.5-Pro	62.6	31.5
Gemini 2.0 Flash	59.2	30.8
Change (pp)	$-3.4$	$-0.7$

Table 4: Two-backbone sensitivity on HM3D v0.1. Changes from the default are in percentage points (pp).

Configuration	SR (%)	SPL (%)	$\Delta$ SR	$\Delta$ SPL
Basic VLM / no memory	67.8	29.3	$-6.6$	$-2.6$
Voxel memory + direct VLM scoring	70.9	30.1	$-3.5$	$-1.8$
HGWM (full)	74.4	31.9	0.0	0.0
Without goal prior	72.1	30.8	$-2.3$	$-1.1$
Without relation refinement	72.1	30.4	$-2.3$	$-1.5$
Without LLM arbitration	72.7	31.1	$-1.7$	$-0.8$

Table 5: Configurations and component ablations on HM3D v0.2. Metrics are percentages; differences are percentage points relative to full HGWM. The first two baselines change the overall configuration. Non-adaptive weights denote a single control, not separate uniform and static experiments.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Condition	Room	Object	Spatial	Semantic	Depth
Any prior-associated object detected	0.25	0.50	0.20	0.03	0.02
Scene quality $Q>0.8$	0.35	0.30	0.25	0.05	0.05
Room score $s_{r}>0.7$	0.45	0.25	0.20	0.05	0.05
Object score $s_{o}>0.7$	0.30	0.40	0.20	0.05	0.05
Otherwise	0.35	0.30	0.20	0.10	0.05

Appendix

Table 6: Fine-feature weights used by the fine-scoring routine, in first-match order. These are implementation settings, not fitted performance results.

Episode length	10	20	40
Memory (MB)	1.2	2.8	5.1
Graph update (ms)	12	18	25
VLM inference (ms)	2300	2300	2300

Appendix

Table 7: Component runtime and memory usage at different episode lengths. VLM timing is per call and does not include the full navigation pipeline.

Configuration	SR(%) $\uparrow$	SPL(%) $\uparrow$
Basic VLM / no memory	67.8	29.3
VLM / voxel memory	70.9	30.1
HGWM / graph matching	74.4	31.9

Appendix

Table 8: Memory configurations on HM3D v0.2 [ 18 ] . The final row changes both memory and scoring; it does not isolate their effects or measure ambiguity robustness.

Configuration	SR (%)	SPL (%)
Full HGWM	74.4	31.9
Without goal prior	72.1	30.8
Without relation refinement	72.1	30.4
Without LLM arbitration	72.7	31.1
With non-adaptive fine weights	73.5	31.3

Appendix

Table 9: Additional HM3D v0.2 component results.

Quantity	Evidence
Full vs. voxel-memory SR/SPL gains	3.5/1.8 pp
Removal of refinement SR/SPL drops	2.3/1.5 pp
Scene–goal matching vs. prompting: SR/SPL gains	+1.5/+1.4 pp

Appendix

Table 10: Memory, refinement, and matching comparisons. Gains and drops are percentage-point differences. The matching row summarizes an additional comparison and is not an absolute SR/SPL pair.

Panel of Figure 5 — Figure 5: Unedited archived views. Top: case 115_853 at steps 0, 3, and 5. Bottom: case 220_890 at steps 0, 20, and 39. Text and action annotations are part of the original images. The examples illustrate how observations evolve during a short and a longer search.

Case	Logged target	Steps	Logs
115_853	sofa	6	6
160_853	tv screen	7	7
199_890	plant	29	29
220_890	plant	40	40
249_890	tv screen	40	40
265_890	toilet	19	19

Appendix

Table 11: Navigation case collection. Counts refer to recorded steps and available text logs.

Object-Goal Navigation

Object-goal navigation studies how robots find and approach requested objects in unfamiliar environments using visual, spatial, and language cues.

See all 86 papers

HRO: Hierarchical Room-to-Object Framework for Zero-Shot Object Goal Navigation with Large Language Models
Jul 12, 2026Luyuan Jia, Yinfeng YuZero-Shot Learning Large Language Model-Based Robot Planning
School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China. · Joint Research Laboratory for Embodied Intelligence, Xinjiang University. · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University.
ReMemNav: Memory-Based Decision Correction and Target Verification for Zero-Shot Object Navigation
Mar 25, 2026Feng Wu, Wei Zuo, Wenliang Yang +3Robot Navigation Vision-Language Navigation
Fudan University · Tongji University
Hierarchical Fast-Slow ReAct Agent for Zero-Shot Object-Goal Navigation
Aug 10, 2026Zhaochen Lan, Zhi Yang, Yuxiang Fu +1Cognitive Architectures for AI Agents Vision-Language Navigation
School of Mechanical Engineering and Automation, Beihang University, Beijing, China · School of Automation, Beijing Institute of Technology, Beijing, China

Jul 12, 2026J/K move · Enter open · S save

HRO: Hierarchical Room-to-Object Framework for Zero-Shot Object Goal Navigation with Large Language Models

Luyuan Jia, Yinfeng Yu

School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China. · Joint Research Laboratory for Embodied Intelligence, Xinjiang University. · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University.

Zero-Shot Learning Large Language Model-Based Robot Planning Object-Goal Navigation

Open paper AskPDF arXiv

Zero-shot object-goal navigation aims to enable an intelligent agent to explore and navigate to objects of unknown categories in an unfamiliar environment without specific target training. In zero-shot navigation tasks, pre-trained large models are usually employed to leverage their prior knowledge for guiding the agent's navigation. However, existing zero-shot object-goal navigation methods based on large language models (LLMs) merely utilize LLMs as flat reasoning tools to directly associate objects or regions. They lack the hierarchical spatial cognition modeling of human-like room semantics to object localization, which leads to strong blindness in exploration, insufficient accuracy in semantic association, and failure to fully unleash the common-sense reasoning potential of LLMs. This paper proposes an LLM-driven hierarchical room-to-object (HRO) framework for zero-shot object-goal navigation, which guides the agent to explore and navigate to the target object in a coarse-to-fine manner. Experiments on Gibson and HM3D datasets verify that our HRO framework achieves superior success rate and generalization over existing LLM-based methods, underscoring LLMs' strong potential for zero-shot object-goal navigation.

Abstract

Object-Goal Navigation

Explore similar work

HRO: Hierarchical Room-to-Object Framework for Zero-Shot Object Goal Navigation with Large Language Models

ReMemNav: Memory-Based Decision Correction and Target Verification for Zero-Shot Object Navigation

Hierarchical Fast-Slow ReAct Agent for Zero-Shot Object-Goal Navigation