NavGPT-3: Harnessing Context in a Hierarchical Navigation Runtime

Gengze Zhou; Yicong Hong; Jiazhao Zhang; Xunyi Zhao; Jian Zhou; Zixing Lei; Zun Wang; Chongyang Zhao

cs.ROOct 7, 2026

NavGPT-3: Harnessing Context in a Hierarchical Navigation Runtime

Authors: Gengze Zhou, Yicong Hong, Jiazhao Zhang, Xunyi Zhao, Jian Zhou, Zixing Lei, Zun Wang, Chongyang Zhao, +4 more

Organizations: AIML, Adelaide University · Metacognition · Roblox · PKU · SJTU · UNC Chapel Hill · UNSW · ANU

Abstract

Language models trained with long-horizon agentic reinforcement learning can generalize knowledge through reasoning, express precise actions, and pursue goals over many steps, raising the ceiling on what an embodied agent can understand and decide. Physical interaction, however, remains the domain of action policies, which provide dense, low-latency control. We present NavGPT-3, a harness that connects the two models, with an OS-like runtime built above it: reasoning, acting, and monitoring run as threads with their own context, tools, and permissions, while the runtime schedules them and decides which thread controls the robot's motion, so that the robot can react to sudden real-world events through interruption and thread switching. Beneath it, our action policy NavGPT VLA, trained on 19.28M examples, allocates visual tokens using codec allocation, in proportion to scene change; its 8B model alone reaches 74.51 SR on R2R-CE and leads RxR-CE with 78.19 SR. With the complete harness, NavGPT-3 sets the state of the art on R2R-CE (81.51 SR) and, for the first time, brings an autonomous agent to human level: on RxR-CE it matches human followers in success (90.43 vs. 90.4 SR) and path fidelity (78.47 vs. 77.7 nDTW) at 1 min 22 s per episode, versus roughly 3 min for a human. We comprehensively ablate the harness design and the interaction between the two models, showing how tools and the action policy shape the path from language-model reasoning to physical control: when NavGPT VLA executes the route, the reasoning loop shortens and the system's minimum reaction time falls from 3-19 s per language-model decision to 0.5-1 s per action-policy step (1-2 Hz). These results show that designing this embodied interface is central to connecting frontier language-model intelligence with low-level physical control. We will release all models, code, and evaluation records.

Figures & tables

Figure 1: Overview of NavGPT-3 : (a) runtime abstraction with the planner, VLA, and spatial tools; (b) synchronous tool use versus asynchronous thread coordination; and (c) benchmark performance. NavGPT-3 reaches human performance on RxR-CE .

Figure 2: Illustrative foreground-control patterns in the NavGPT-3 Runtime. Thin bars mark concurrent activity; the teal path denotes motion authority. (A) A route-review interrupt pauses main execution for correction, followed by resumption. (B) A safety interrupt revokes authority while inference continues; resumption requires fresh state and runtime authorization.

Figure 3: Codec-weighted visual-context allocation. (a) Change, recency, and view jointly determine whole-image resolution for a four-view history, with an initial I-frame and subsequent P-frames. (b) Optical-flow diagnostics and resulting token grids for the four views at $t_{16}$ . (c) Allocation change relative to positional weighting versus RGB change $100\delta_{h,v}$ for 41,216 replayed image/context pairs from 13 recorded histories at $B_{\mathrm{vis}}=3072$ . Curve: local regression; numbers match (b).

Figure 4: NavGPT VLA training mixture. Supervision split and effective records by data family.

Family	Sources	Sampling	Size (M)
Action regression
VLN: four-view	7	100%	3.938
VLN: single-view	5	100%	2.113
VLN: SRDF	2	15%	0.516
Point-goal navigation	5	100%	0.984
Object-goal navigation	1	100%	2.000

Table 1: Training-data mixture. Sizes are effective records per epoch, in millions.

Backbone	GPUs	BS	Steps	Length	LR	AH LR
Qwen3-VL-4B	32	256	$\sim$ 75.3k	8,192	$5{\times}10^{-6}$	$10^{-3}$
Qwen3-VL-8B	32	256	$\sim$ 75.3k	8,192	$5{\times}10^{-6}$	$10^{-3}$

Table 2: Matched VLA training configurations. BS: global batch size; AH LR: action-head learning rate.

Figure 5: Harness development through recursive self-improvement: the research loop, candidate revisions, and success-rate progression.

Planner	Tools			Navigation				Resource use
Planner	VLA	Map	Backtrack	NE $\downarrow$	OSR $\uparrow$	SR $\uparrow$	SPL $\uparrow$	Tokens (k) $\downarrow$	Turns $\downarrow$	Cost ( $)$ \downarrow$	Infer. (s) $\downarrow$	React. (s) $\downarrow$
VLA only	✓	–	–	4.23	78	71	66.14	–	–	–	44.1	0.78
GPT-6 Astra	–	–	–	4.67	74	72	62.26	325.0	32.14	0.593	105.7	3.3
GPT-6 Astra	–	✓	✓	2.55	87	83	69.04	190.8	10.71	0.469	66.8	6.2
GPT-6 Astra	✓	–	–	4.58	79	75	67.83	227.9	3.37	0.469	84.4	0.78
GPT-6 Astra	✓	✓	–	4.01	79	76	67.35	201.4	3.42	0.443	80.4	0.78

Table 3: Tool and VLA ablation on the R2R-CE subset. Resource use is averaged per episode. Opus costs are as reported by the provider’s SDK; Astra costs are estimated from recorded usage at list prices. Inference excludes communication and harness overhead. Reaction is the mean interval between environment actions: Planner time per turn without the VLA, VLA inference per step with it.

Group	Tool	Effect on state	Returned to the Planner
Basic	observe_forward()	None	Current forward RGB for fine semantic inspection.
	observe_panorama()	Updates observed local coverage, not pose	Four labelled views with bearings, clearance, pose displacement, and node information.
	navigate_relative()	Executes a validated local turn and translation	Realized motion status, endpoint panorama, and nearby node candidates.
	terminate_episode()	Optionally returns to a selected node, then stops	Stop or return status; this call ends the episode and cannot be undone.
VLA	navigate_by_ instruction()	Runs one VLA rollout; the VLA keeps its history	Up to eight sampled route keyframes, endpoint panorama, optional observed map, and execution status.
Map	observe_map()	Updates observed coverage, not pose	Observed-only top-down map, map status, and route-ordered node table.

Table 4: Tools that implement the navigation capabilities, grouped as in the ablation of Table 3 .

Figure 6: Visual prompt organization. SR/SPL and token usage for Astra and Opus 5.

Frequency	NE $\downarrow$	OSR $\uparrow$	nDTW $\uparrow$	SR $\uparrow$	SPL $\uparrow$	Tokens (k) $\downarrow$
None	3.82	81	16.74	77	67.51	179.2
32	3.56	81	16.66	78	68.88	1,529.9
16	2.01	85	18.72	83	77.36	2,319.5

Table 5: Check frequency. VLA steps per check.

Method	Model	R2R-CE Val-Unseen				RxR-CE Val-Unseen
Method	Model	NE $\downarrow$	OSR $\uparrow$	SR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	nDTW $\uparrow$	SR $\uparrow$	SPL $\uparrow$
Human † ( Ku et al., 2020 )	–	–	–	–	–	1.32	77.7	90.4	–
NavFoM ( Zhang et al., 2025a )	7B	4.61	72.1	61.7	55.3	4.74	65.8	64.4	56.2
ABot-N0 ( AMAP CV Lab, 2026 )	4B	3.78	70.8	66.4	63.9	3.83	–	69.3	60.0
AstraNav-World ( Chen et al., 2026 )	3B+5B	3.86	73.9	67.9	65.4	3.82	–	72.9	61.5
OmniNav ( Xue et al., 2025 )	3B	3.74	74.6	69.5	66.1	3.77	–	73.6	62.0

Table 6: Vision-and-language navigation. Evaluation results on full R2R-CE and RxR-CE Val-Unseen splits. † RxR Val-Unseen English human followers ( Ku et al., 2020 ) , measured in DE. Both NavGPT-3 systems use the 8B VLA.

Table 13

Figure 7: Codec comparison across training scopes.

Figure 8: Model and data scaling.

Figure 9: Recorded glass-door execution. The outlined fourth panel highlights the near-door forward pause, followed by lateral correction and continued passage.

Scenario	Controlled event	Primary measurements
A: Route revision	A detour or endpoint correction becomes necessary during an extended rollout; vary event timing relative to Planner inference.	Completion SR; event-to-authorized-repair time; route efficiency; Planner calls; execution idle time.
B: LiDAR protection	A soft obstacle enters a calibrated protected region during motion, including while the Planner is busy. Use matched harmless passages as negatives.	Missed and false stops; trigger-to-halt p50/p95; post-trigger distance; minimum clearance; operator interventions.
C: Priority and recovery	Trigger a hazard during repair or near VLA completion; delay a model reply, resend an outdated command to the robot’s command filter, then clear the hazard.	Accepted stale commands; event ordering; halt latency; unauthorized resumption; correct resumption and task completion.

Table 9: Go2 scenarios and measurements.

	A: Revision		B: Protection		C: Recovery
Condition	SR	Repair (s)	Missed/ $N$	Halt p95 (ms)	Stale/ $N$	Resume SR
VLA-only	60.0	–	–	–	–	–
Planner + tools	70.0	14.2	9/30	2150	11/30	53.3
Sequential + queued	73.3	12.8	7/30	1840	8/30	60.0
Sequential + priority	76.7	11.5	2/30	420	3/30	70.0
Overlapping + queued	76.7	9.6	5/30	1310	6/30	66.7

Table 10: Go2 runtime results.

Figure 10: Backtracking to an overshot endpoint. Claude Opus 5 on R2R-CE, episode 144. The Planner identifies the hallway table before the VLA’s bedroom stop, retraces 8.54 m to node 1, and verifies a 1.2 m local correction before stopping.

Figure 11: Completing omitted route clauses. Claude Opus 5 on RxR-CE, episode 2523. The Planner returns from a dead-end nook to node 8, then uses relative motion to reach the gray cabinet and the open doorway with a rising staircase.

Figure 12: Recovering from a blocked aisle. GPT-6 Astra on R2R-CE, episode 43. After repeated VLA execution and a direct motion attempt stall, the Planner returns to node 3, traverses the other aisle, and stops beyond the table beside the lamp.

Figure 13: Resuming delegation after unsuccessful local correction. Claude Opus 5 on RxR-CE, episode 5782. With the stair clause still unfinished, the Planner delegates the full instruction again. The second rollout reaches the upstairs bathroom, which the Planner verifies before stopping.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Responsibility	Component and role
Thread coordination	Runtime: schedule threads, manage permissions, and control motion authority.
Navigation decisions	Planner: interpret evidence, choose tools, and decide when the goal is reached.
Context construction	Harness: select and render the images and state shown to the models.
Persistent navigation state	Harness: keep route references, nodes, and annotations.
Tool checking	Harness: validate requests and execute permitted tool calls.
Low-level navigation	VLA: predict waypoints from its visual history.

Appendix

Figure 14: Division of responsibilities in the NavGPT-3 system.

Capability	Navigation purpose	What it takes and returns
Local observation	Ground landmarks and inspect the current surroundings	Relate forward or panoramic appearance to the current viewing direction and pose.
Instruction delegation	Execute an extended route with the learned policy	Pass the requested instruction to the VLA, keep its history across calls, and return images of the executed route with its status.
Route inspection	Relate current progress to previously encountered places	Show the observed map and the ordered list of visited places, without marking unobserved space as free.
Spatial revision	Revisit a recorded place or correct a local endpoint error	Turn a reference to a visited place, or a relative motion request, into checked movement and report what was executed.
Semantic annotation	Retain an interpretation of a visited place	Attach a label to a recorded place without changing its recorded position.
Task termination	Accept the final navigation state	Make episode completion explicit and distinct from the end of a delegated rollout.

Appendix

Table 11: Navigation capabilities and their interfaces.

Operation	Planner	VLA	Enforced by
Interrupt the running VLA rollout	May request, if permitted	Stops after the current step	Runtime: revokes the permission to move
Request an early Planner review	Receives it, if permitted	Keeps executing	Runtime: queues or discards the request
Resume or revise the route	Chooses an available tool	Executes a valid delegation	Runtime: grants one operation the permission to move
Change physical or route state	Cannot change it directly	Proposes waypoints	Environment: executes motion; harness: updates route records
End the episode	Decides explicitly to stop	Its stop ends only the delegation	Runtime: enforces the stop and episode limits
Share information between Planner and VLA	Sends structured instructions only	Returns structured status and route evidence only	Runtime: controls visibility; harness: builds context

Appendix

Table 12: What each component may do when execution is interrupted.

$B_{\mathrm{vis}}$	$N_{\mathrm{hist}}$	$N_{\mathrm{view}}$	$b_{\max}$	$\rho_{\mathrm{vis}}$	Images differing	Mean $\|\Delta b\|$ (tokens)
3072	2	1	196	7.84	0 / 2	0.0
3072	8	1	196	1.96	0 / 8	0.0
3072	15	1	196	1.04	0 / 15	0.0
4096	4	4	256	1.00	0 / 16	0.0
3072	16	1	196	0.98	3 / 16	8.0
3072	8	4	196	0.49	30 / 32	47.3

Appendix

Table 13: Allocation saturation. Deterministic synthetic change-score diagnostic.

Setting	Value
RGB resolution	512 pixels
Camera order	Front, right, back, left
Returned route keyframes	At most 8, uniformly sampled along the rollout
Episode / delegation limit	500 / 200 environment steps
Planner-call limit	200 turns per episode
Episode timeout	2,400 s

Appendix

Table 14: Harness evaluation settings.

Statistic	Full	Subset	$\|\mathrm{SMD}\|$	$D$
Episodes	1,839	100	–	–
Scenes	11	10	–	–
Unique reference trajectories	613	97	–	–
Geodesic distance (m)	$8.90\pm 2.67$	$8.99\pm 3.06$	0.033	0.072
Instruction length (words)	$26.65\pm 11.48$	$24.68\pm 8.28$	0.207	0.103
Directional mentions	$2.14\pm 1.80$	$2.20\pm 1.75$	0.034	0.040

Appendix

Table 15: R2R ablation-subset coverage and distributions.

Dataset	System	$N$	SR	SPL	$\Delta$ SR vs. full
MP3D	Qwen-RobotNav 8B	500	47.8	17.5	$-1.0$
MP3D	NavGPT-3	500	59.2	25.2	–
HM3D v2	Qwen-RobotNav 8B	500	72.4	33.1	$+1.2$
HM3D v2	NavGPT-3	500	75.3	40.8	–

Appendix

Table 16: ObjectNav comparison on identical episodes.

Judge	Labeled $N$	Agreement (%)
GPT-4o-mini	50	72.0
GPT-6 Astra	50	92.0
Astra $-$ 4o-mini		+20.0

Appendix

Table 17: EXPRESS judge agreement with human labels.

Explore similar work

Uni-LaViRA: Language-Vision-Robot Actions Translation for Unified Embodied Navigation
May 26, 2026Hongyu Ding, Sizhuo Zhang, Ziming Xu +13Large Language Model-Based Robot Planning Robot Navigation
Nanjing University, Nanjing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · Beihang University, Beijing, China +2
C $^2$ Nav: Compare Before You Commit for Zero-Shot Vision-and-Language Navigation
Sep 14, 2026Runtian Zheng, Congpeng Zhang, Ying LiuVision-Language Navigation VLM Reasoning
North China University of Technology, Beijing, China
GroundingVLN: Reasoning and Acting with Grounding for Vision-Language Navigation
Sep 16, 2026Kailing Li, Yu Han, Tianwen Qian +4Vision-Language Navigation Vision-Language Grounding
School of Computer Science and Technology, East China Normal University · NeoteAI · King Abdullah University of Science and Technology

May 26, 2026, cs.ROJ/K move · Enter open · S save

Uni-LaViRA: Language-Vision-Robot Actions Translation for Unified Embodied Navigation

Hongyu Ding, Sizhuo Zhang, Ziming Xu, Jinwen Guo+12

Nanjing University, Nanjing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · Beihang University, Beijing, China+2

Large Language Model-Based Robot Planning Robot Navigation Vision-Language Navigation Multimodal Agents

Open paper AskPDF arXiv

Embodied navigation requires an agent to map language and visual observations to a stream of spatial actions that drive a real robot through environments it has never seen. The dominant approach has been to scale vision-language-action (VLA) foundation models on ever-larger collections of robot trajectories. This paper argues that, for navigation specifically, generality can be obtained structurally, not only through data scale. The underlying decision structure of navigation reduces to a single Language-Vision-Robot Actions Translation. The language action emits semantic-level directional command and the vision action emits a pixel-level visual target. Both outputs lie inside the natural output manifold of pretrained multimodal large language models (MLLMs), so the task can be reasoned about by an agent rather than learned from robot data. Therefore, we present Uni-LaViRA, a unified agentic architecture that extends the same insight to four task families (VLN-CE, ObjectNav, EQA, and Aerial-VLN) and to four heterogeneous real robots (Wheeled, Quadruped, Humanoid robot, and a self-built UAV) in a zero-shot manner. Two agent-loop mechanisms make this unification practical. TODO List Memory (TDM) rewrites a structured checklist of pending sub-goals at every step, reciting the unfinished items back into the agent's most recent attention window. Second Chance Backtrack (SCB) rolls the robot back to the pre-error state and conditions the agent's next plan on the failed sub-trajectory, turning single-pass navigation into a self-correcting process. With zero training effort, Uni-LaViRA reaches 60.7% SR on VLN-CE R2R, 51.3% on VLN-CE RxR, 77.7% on HM3D-v2, 60.0% on HM3D-OVON, 54.7% on MP3D-EQA, and 40.0% on OpenUAV, matching or even surpassing recent training navigation foundation models that consume millions of samples and thousands of GPU-hours.

Abstract

Explore similar work

Uni-LaViRA: Language-Vision-Robot Actions Translation for Unified Embodied Navigation

C2^22Nav: Compare Before You Commit for Zero-Shot Vision-and-Language Navigation

GroundingVLN: Reasoning and Acting with Grounding for Vision-Language Navigation

C $^2$ Nav: Compare Before You Commit for Zero-Shot Vision-and-Language Navigation