Organizations: Australian Institute for Machine Learning, Adelaide University · The University of Manchester · Zhejiang University · Agency for Science, Technology and Research (A*STAR)
A navigable agent needs to understand both high-level semantic instructions and precise spatial perceptions. Building navigation agents centered on Multimodal Large Language Models (MLLMs) demonstrates a promising solution due to their powerful generalization ability. However, the current tightly coupled design dramatically limits system performance. In this work, we propose a decoupled design that separates low-level spatial state estimation from high-level semantic planning. Unlike previous methods that rely on predefined, oversimplified textual maps, we introduce an interactive metric world representation that maintains rich and consistent information, allowing MLLMs to interact with and reason on it for decision-making. Furthermore, counterfactual reasoning is introduced to further elicit MLLMs' capacity, while the metric world representation ensures the physical validity of the produced actions. We conduct comprehensive experiments in both simulated and real-world environments. Our method establishes a new zero-shot state-of-the-art, achieving 48.8% Success Rate (SR) in R2R-CE and 42.2% in RxR-CE benchmarks. Furthermore, to validate the versatility of our metric representation, we demonstrate zero-shot sim-to-real transfer across diverse embodiments, including a wheeled TurtleBot 4 and a custom-built aerial drone. These real-world deployments verify that our decoupled framework serves as a robust, domain-invariant interface for embodied Vision-and-Language navigation.
Figures & tables
Fig. 1: Bridging high-level reasoning and embodied execution with GTA. Left : Existing MLLM-based VLN agents reduce the rich 3D environment into an oversimplified linearized text memory according to human knowledge. Right : Our GTA framework decouples spatial modeling from semantic reasoning. Our Interactive Metric World Representation maintains rich spatial and historical information, enabling the MLLM to interact with it accordingly for decision-making. This also enables Counterfactual Reasoning, which further elicits the MLLM’s capacity.
Fig. 2: Qualitative visualization of a navigation episode in R2R-CE. The top row displays the agent’s Metric World Representation (top-down metric view), while the bottom row shows the corresponding egocentric panoramic observations at each step. The planned trajectory is marked in yellow, with blue dots indicating waypoints and the red arrow showing the agent’s current pose.
Fig. 3: Overview of the GTA Framework. Our architecture decouples spatial modeling from semantic reasoning. The Metric Mapping Module (left) fuses sparse RGB-D streams via TSDF reconstruction to synthesize a real-time metric map. We construct the Interactive Metric World Representation by unifying this geometric reconstruction with Procedur al Reasoning Blueprints, which comprise the logical “TODO List” and topological history. This composite spatial-logic state is rendered via our Interactive Reasoning Interface into a structured prompt for Counterfactual Reasoning Brain driven by the frozen MLLM (right). Leveraging this unified context, the MLLM directly infers the next metric waypoint (x,y,z)
#
Methods
R2R-CE
RxR-CE
NE ↓
OSR ↑
SR ↑
SPL ↑
nDTW ↑
NE ↓
SR ↑
SPL ↑
nDTW ↑
Supervised Learning:
1
CMA [ 33 ]
6.30
49.0
38.0
33.0
–
10.4
24.1
19.1
37.4
2
VLN-BERT [ 33 ]
5.74
53.0
44.0
39.0
–
8.98
27.1
22.7
46.7
3
GridMM [ 11 ]
5.11
61.0
49.0
41.0
–
–
–
–
–
4
ETPNav [ 22 ]
4.71
65.0
57.0
49.0
–
5.64
54.8
44.9
61.9
TABLE I: Comparison in continuous environments on R2R-CE and RxR-CE Val-Unseen splits. The best supervised results are highlighted in bold , while the best zero-shot results are underlined . We report metrics on the full validation split for R2R-CE (1839 episodes) and a sampled subset for RxR-CE (260 episodes).
Method - DE
R2R-Sampled
REVERIE-Sampled
OSR(↑)
SR(↑)
SPL(↑)
OSR(↑)
SR(↑)
SPL(↑)
NavGPT
53.2
43.5
34.7
37.2
31.2
23.4
NavGPT + EWR
55.6
45.0
37.2
41.7
36.1
26.8
Method - CE
R2R-CE-Sampled
REVERIE-CE-Sampled
OSR(↑)
SR(↑)
SPL(↑)
OSR(↑)
SR(↑)
SPL(↑)
OpenNav
33.3
30.6
23.5
22.8
16.1
11.1
TABLE II: Comparison with baselines enhanced with Explicit World Representation (EWR). To ensure environmental diversity, we sample 180 trajectories from the discrete R2R and REVERIE datasets. In continuous environments, specifically for R2R-CE , we curate a challenging subset of 180 episodes defined in Sec. V-A .
Method
R2R-Val-Sampled
NE ( ↓ )
OSR ( ↑ )
SR ( ↑ )
SPL ( ↑ )
nDTW ( ↑ )
GTA (w/o PB)
5.38
48.3
45.0
38.1
52.7
GTA (w/ PB)
5.27
56.7
47.2
39.6
55.8
TABLE III: Ablation study on Procedural Reasoning Blueprints (PB). We evaluate the impact of decomposing instructions into logical blueprints on the curated R2R-CE subset (180 episodes). ↑ denotes higher is better. Bold numbers indicate the best performance.
Method - DE
R2R-Val-Sampled
NE ( ↓ )
OSR ( ↑ )
SR ( ↑ )
SPL ( ↑ )
nDTW ( ↑ )
GTA (Qwen3-VL-235B)
5.98
46.1
37.2
29.5
51.3
GTA (Gemini-2.5 Pro)
5.35
48.7
42.3
34.4
56.2
GTA (GPT 5.1)
5.27
56.7
47.2
39.6
55.8
TABLE IV: Comparison of different MLLM backbones on the R2R-CE dataset. Evaluations are conducted on the same curated subset of 180 challenging trajectories defined in Sec. V-A . ↑ denotes higher is better, and ↓ denotes lower is better. Bold numbers indicate the best performance.
Fig. 4: Zero-shot Sim-to-Real transfer across diverse robot embodiments. We demonstrate the generalization capability of GTA by deploying it on two distinct physical platforms in unseen real-world environments. Top Rows (Wheeled Robot): A TurtleBot 4 successfully executes instructions requiring obstacle negotiation and semantic grounding of large objects. Bottom Row (Drone): A custom-built aerial vehicle utilizes the same framework to locate a fine-grained target.
Method
SR ↑ (%)
NE ↓ (m)
Supervised Learning:
VLN-BERT [ 33 ]
16.0
5.36
RDP [ 41 ]
20.0
5.45
Zero-Shot Learning:
SmartWay [ 12 ]
32.0
4.85
GTA - Wheeled Robot (Ours)
40.0
3.66
TABLE V: Real-world navigation performance. We compare GTA against representative supervised (VLN-BERT, RDP) and zero-shot (SmartWay) baselines. ↑ indicates higher is better, ↓ indicates lower is better.
School of Information Science and Electronic Engineering & School of Integrated Circuits, Shanghai Jiao Tong University · Institute of Artificial Intelligence, China Telecom · 3Central South University +1