Organizations: GRASP Laboratory, University of Pennsylvania, Philadelphia, PA 19104, USA · Department of Electrical and Systems Engineering, University of Pennsylvania, Philadelphia, PA 19104, USA
Learned navigation policies typically consume observations as a temporally ordered history, with positional encodings tying each observation to when it was seen, making it difficult to reuse experience from earlier traversals of an environment. Systems that do reuse such experience usually construct an explicit representation, such as a map or a topological graph, and plan on it. We propose VNT-PA (Visual Navigation Transformer with Pose Attention), a transformer planner whose context is a set of depth keyframes indexed by camera pose. With camera poses as positional encoding, attention depends on the pose differences between keyframes rather than on their temporal order. VNT-PA is trained to imitate a shortest-path planner operating on the ground-truth scene mesh, predicting actions by querying the spatial context with only its current pose and the goal position. On point-goal navigation in HM3D validation scenes, VNT-PA reaches 93.3% success and 90.4% success weighted by path length (SPL), outperforming baselines that encode the same context as a temporal sequence or treat pose as an input feature, in both navigation performance and training efficiency. Because the spatial context is a pose-indexed set, frames from different trajectories can be fused at test time. The planner also degrades more gracefully under localization noise than a conventional baseline which plans on explicit maps. These results show that pose-stamped experience can serve directly as the environment representation for a learned planner, and that making attention depend on pose differences, rather than on temporal order, speeds up training and improves long-horizon navigation.
Figures & tables
Fig. 1: Overview of VNT-PA. (1) A keyframe filter keeps only depth frames that observe sufficient new space. A frozen depth encoder (DeFM) and a learned attention-pooling summarizer represent each keyframe as a single token ei . (2) The set formed by image tokens indexed by corresponding camera poses is referred to as spatial context. It serves as an implicit representation of the environment. Together with a learned goal token egoal stamped at the goal position pgoal , the context set passes through Lmap bidirectional self-attention blocks that use poses as rotary positional embeddings. The output embedding is referred to as goal-aware spatial context. (3) A query token equery is built from the relative goal location and stamped at the robot pose probot . It cross-attends the goal-aware context through Lread cross-attention blocks with the same positional embedding. (4) An autoregressive GRU decodes the final query token embedding into a chunk of actions. The robot executes the first h actions and then queries the planner from its new pose.
Detour ratio κ
Geodesic distance (m)
Overall
<1.2
[1.2,1.5)
≥1.5
<10
[10,15)
≥15
Method
SR
SPL
SR
SPL
SR
SPL
SR
SPL
SR
SPL
SR
SPL
SR
SPL
LSTM [ 33 ]
31.9 ± 0.6
28.5 ± 0.6
43.1 ± 1.3
38.6 ± 1.1
29.3 ± 0.7
26.2 ± 0.7
11.3 ± 0.8
9.6 ± 0.7
40.3 ± 0.7
35.9 ± 0.6
15.8 ± 0.6
14.4 ± 0.6
5.7 ± 0.3
5.1 ± 0.3
SRU [ 22 ]
30.3 ± 1.5
27.3 ± 1.4
43.0 ± 1.6
38.9 ± 1.3
24.8 ± 1.2
22.3 ± 1.3
11.4 ± 1.8
10.0 ± 1.6
38.2 ± 1.5
34.4 ± 1.4
14.4 ± 1.8
13.1 ± 1.8
7.8 ± 1.1
7.1 ± 1.1
Causal Transformer
90.3 ± 0.5
87.3 ± 0.5
94.1 ± 0.7
91.0 ± 0.5
88.7 ± 0.8
85.9 ± 0.8
84.5 ± 0.8
81.2 ± 0.8
92.8 ± 0.5
89.8 ± 0.5
86.4 ± 0.7
83.2 ± 0.6
80.4 ± 1.7
77.4 ± 1.3
VNT-TA
87.1 ± 3.5
83.9 ± 3.8
92.0 ± 2.0
88.7 ± 2.2
85.9 ± 4.6
82.7 ± 5.0
78.3 ± 5.0
75.2 ± 5.3
91.0 ± 3.0
87.8 ± 3.4
82.3 ± 4.6
79.2 ± 4.7
67.7 ± 4.3
64.9 ± 4.4
TABLE I: Closed-loop navigation performance with ground truth localization. Mean ± standard deviation over three training seeds.
Fig. 2: Closed-loop navigation of VNT-PA in unseen evaluation scenes. White marks the navigable space of the ground-truth navigation mesh, and blue markers show the keyframe poses in the spatial context. The trajectory of VNT-PA is colored from dark at the start (circle) to bright at the goal (star), and the dashed line shows the expert trajectory.
Spatial context
Overall
SR by geodesic distance (m)
Exploration frames
Robot views
SR
SPL
<10
[10,15)
≥15
Keyframes
94.6
91.7
96.2
92.6
87.7
Keyframes
✓
96.5
93.5
97.5
94.0
95.2
All frames
96.2
93.1
97.4
94.4
90.9
All frames
✓
97.1
94.0
98.0
94.6
96.3
TABLE II: Closed-loop navigation of VNT-PA (single seed) with its spatial context extended at test time.
Fig. 3: Closed-loop SPL of intermediate checkpoints, mean ± standard deviation over models trained with three random seeds.
Exact poses
Noisy poses
Context
Method
SR
SPL
SR
SPL
Keyframes
OctoMap + A*
96.2
92.2
54.1
46.0
VNT-PA
94.6
91.7
86.6
82.7
All frames
OctoMap + A*
96.1
92.1
78.1
72.4
VNT-PA
96.2
93.1
88.0
84.1
TABLE III: Closed-loop navigation of VNT-PA (single seed) and conventional pipeline with exact and noisy poses
Fig. 4: Summarizer attention pattern on two sampled frames from evaluation scenes. Left: RGB images, for reference only. Right: attention of the summarizer’s learned query token over patch tokens, averaged over attention heads and overlaid on the depth image.
Vision-Language Navigation (VLN) requires long-horizon instruction-consistent planning and short-horizon local planning in current geometry. Video-action models (VAMs) offer a promising way to learn predictive representations for navigation. However, existing VAMs jointly predict visual futures and local trajectories in a single process, even though they operate at different paces: future prediction remains stable across multiple updates while local trajectories must react to fresh observations. To bridge this gap, we introduce HAVANA, a dual-system VAM that decouples predictive reasoning from reactive local planning. A slow system produces a compact language-conditioned latent plan from observation history, while a fast system combines the latest plan with the current observation to generate local trajectories. Their asynchronous execution enables rapid adaptation without repeated future prediction, and no future video is generated during inference. Experiments on OmniNavBench demonstrate state-of-the-art navigation performance over prior baselines, while real-world deployments further show robust generalization to unseen environments across different robot embodiments. Our code and checkpoints will be available at https://github.com/Anastasiawd/HAVANA.
Dongzhihan Wang, Yi Du, Chen Wang +2
Tongji University · SAIR Lab · Shanghai University
What visual representation should a navigation policy act on? Navigation policies built on vision foundation models largely leave representation design to the encoders' pretraining objectives. We present RepresentNav, a navigation foundation model built on a self-constructed representation space: a visual representation constructed by the policy's own objectives from vision foundation models. RepresentNav builds this space with Representation Residuals, a compact three-layer transformer that attends over features spanning the full depth of multiple complementary vision foundation models, so that the policy learns to see what it needs for navigation. In the same space it dreams what it wants: before every action a latent chain-of-thought predicts the representation of the scene a moment ahead. The dream is trained end-to-end with the policy as a joint-embedding predictive architecture (JEPA), with a next-representation prediction loss and a regularizer on its targets. RepresentNav sets a new state of the art on VLNVerse, VLN-CE R2R, EVT-Bench, and two point-goal benchmarks, and its gains carry over to a physical robot amid clutter and moving people. On VLNVerse fine-grained, RepresentNav achieves 83.80% success, outperforming the previous state of the art by 20.05 percentage points, and on EVT-Bench distracted tracking it reaches 85.5% with a single camera (+10.1). With the same encoders, the self-constructed space clearly outperforms concatenating their outputs and learns to weight their layers unevenly, and a dream in it cuts collisions by a quarter. A single dreamed latent decodes into the depth, semantics, and 3D occupancy of the scene a moment ahead, more accurately than pretrained encoders applied to the real future frame, although occupancy is never supervised, and the rollout tracks the robot's own motion over the next second: the dream emerges as a compact latent world model.
Jiawei Chen, Yinxi Wang, Zihan Yang +12
Peking University · University College London, University of London · DataCanvas
Visual navigation policies have grown to hundreds of millions of parameters trained on billions of frames, with geometry, mapping, and control learned implicitly. We propose a decomposed point-goal navigation system in which operations with known closed-form structure, such as projective geometry, occupancy, and coordinate transforms, are computed analytically and serve as interfaces between three small learned modules: an egress predictor that grounds the episode goal as a local subgoal in the current view, a navigation predictor that estimates a goal-conditioned posterior over where trajectories travel, and an endpoint-pinned residual diffusion generator that samples trajectory shapes from this posterior. Only 0.58M out of 23M parameters are trained, on 44k frames, in under one GPU-hour. Across 6060 point-goal episodes in 60 environments, the system attains competitive success rates with the lowest collision rate among evaluated methods. We further show that under this decomposition, the frozen image encoder can be replaced by a 0.54M MobileNetV2 at a -2.0 SR cost, bringing the full system under 1.2M parameters. It also transfers to no-goal exploration by retraining only the 123k-parameter egress head, and its failure modes under sensor corruption are transparent and analytically correctable. We deploy and evaluate the system zero-shot on a low-cost UGV, running navigation and localization on a Jetson Orin Nano in real-time.
Edward Beng Wai Tan, Siew-Kei Lam
College of Computing and Data Science, Nanyang Technological University, Singapore.