Humans routinely leverage semantic hints provided by signage to navigate to destinations within novel Large-Scale Indoor (LSI) environments, such as hospitals and airport terminals. However, this capability remains underexplored within the field of embodied navigation. This paper introduces a novel embodied navigation task, SignNav, which requires the agent to interpret semantic hint from signage and reason about the subsequent action based on current observation. To facilitate research in this domain, we construct the LSI-Dataset for the training and evaluation of various SignNav agents. Dynamically changing semantic hints and sparse placement of signage in LSI environments present significant challenges to the SignNav task. To address these challenges, we propose the Spatial-Temporal Aware Transformer (START) model for end-to-end decision-making. The spatial-aware module grounds the semantic hint of signage into physical world, while the temporal-aware module captures long-range dependencies between historical states and current observation. Leveraging a two-stage training strategy with Dataset Aggregation (DAgger), our approach achieves state-of-the-art performance, recording an 80% Success Rate (SR) and 0.74 NDTW on val-unseen split. Real-world deployment further demonstrates the practicality of our method in physical environment without pre-built map.
Figures & tables
Fig. 1: Capital Region International Airport in Clinton County, Michigan . When humans first arrive at an airport terminal, they routinely leverage the semantic hints of signage to find their boarding gates. We formulate this problem as an embodied navigation task SignNav and build the LSI-Dataset to promote the solution of SignNav.
Fig. 2: An episode example in a hospital scene. The agent is required to make action decisions according to the semantic hints of signage (detected directional arrows) and finally arrive at the target location.
Fig. 3: The architecture of our Spatial-Temporal Aware Transformer (START) model for the SignNav task. START uses a spatial-aware module for grounding the semantic hint of sigange into egocentric observation, and a temporal-aware module for capturing long-range dependencies between the historical states and current observation.
Fig. 4: Different action decisions under different circumstances. (a) different semantic hints at the same location; (b) the same semantic hints at different locations; (c) different semantic hints at different locations.
Fig. 5: One period in an episode that there is no visible semantic hints of signage. The agent still needs to make correct decisions when there is no visible semantic hint.
Fig. 6: Trajectories visualization of our method compared to baselines. The second row shows the first-person-view of agent as it navigates along the ground-truth trajectory.
Methods
Val-Seen
Val-Unseen
SR
NDTW
SDTW
SR
NDTW
SDTW
VLM-based
0.28
0.46
0.21
0.30
0.54
0.25
Rule-based [ 28 ]
0.51
0.65
0.42
0.67
0.73
0.56
ViNT [ 39 ]
0.76
0.71
0.64
0.68
0.69
0.57
START
0.90
0.80
0.76
0.80
0.74
0.67
TABLE I: Performance comparison with different baselines.
RMSE (m)
Navigation Distance (m)
40
80
120
160
200
Val-Seen
0.28
0.25
0.24
0.25
0.24
Val-Unseen
0.25
0.26
0.26
0.28
0.27
TABLE II: Trajectory error statistics at different distances.
Fig. 7: Real-world deployment of our method in an office building to find the “Lifts” leveraging signage.
Spa- Aware
Tem- Aware
Val-Seen
Val-Unseen
SR
NDTW
SDTW
SR
NDTW
SDTW
×
✓
0.60
0.67
0.50
0.39
0.57
0.33
✓
×
0.61
0.66
0.51
0.47
0.59
0.39
✓
✓
0.82
0.76
0.69
0.72
0.72
0.60
TABLE III: Ablation study for spatial and temporal-aware modules.
Fig. 8: Attention map visualization both with the spatial-aware and temporal-aware modules in our START model. The darker the color, the greater the attention weight.
DAgger
Val-Seen
Val-Unseen
SR
NDTW
SDTW
SR
NDTW
SDTW
×
0.82
0.76
0.69
0.72
0.72
0.60
✓
0.90
0.80
0.76
0.80
0.74
0.67
TABLE IV: Ablation study for the impact of training with DAgger.
RGB
Depth
Val-Seen
Val-Unseen
SR
NDTW
SDTW
SR
NDTW
SDTW
✓
×
0.67
0.70
0.56
0.59
0.68
0.49
×
✓
0.86
0.78
0.73
0.77
0.73
0.64
✓
✓
0.82
0.76
0.69
0.72
0.72
0.60
TABLE V: Ablation study for the impact of different modal inputs.
Recent embodied navigation approaches leveraging Vision-Language Models (VLMs) demonstrate strong generalization in versatile Vision-Language Navigation (VLN). However, reliable path planning in complex environments remains challenging due to insufficient spatial awareness. In this work, we introduce SPAN-Nav, an end-to-end foundation model designed to infuse embodied navigation with universal 3D spatial awareness using RGB video streams. SPAN-Nav extracts spatial priors across diverse scenes through an occupancy prediction task on extensive indoor and outdoor environments. To mitigate the computational burden, we introduce a compact representation for spatial priors, finding that a single token is sufficient to encapsulate the coarse-grained cues essential for navigation tasks. Furthermore, inspired by the Chain-of-Thought (CoT) mechanism, SPAN-Nav utilizes this single spatial token to explicitly inject spatial cues into action reasoning through an end-to end framework. Leveraging multi-task co-training, SPAN-Nav captures task-adaptive cues from generalized spatial priors, enabling robust spatial awareness to generalize even to the task lacking explicit spatial supervision. To support comprehensive spatial learning, we present a massive dataset of 4.2 million occupancy annotations that covers both indoor and outdoor scenes across multi-type navigation tasks. SPAN-Nav achieves state-of-the-art performance across three benchmarks spanning diverse scenarios and varied navigation tasks. Finally, real-world experiments validate the robust generalization and practical reliability of our approach across complex physical scenarios.
Robots navigating open-world, human-centered environments must look beyond prebuilt high-fidelity maps and leverage navigational aids embedded in their surroundings, particularly, signs. To advance sign-guided visual navigation, we introduce SiGNgapore, an interactive, multimodal dataset designed to benchmark sign-guided multi-step navigational decision-making. SiGNgapore covers diverse public spaces in Singapore, including shopping malls, public transportation hubs, and hospitals, featuring multi-floor, multi-building layouts and indoor-outdoor transitions. It consists of 56 long-horizon navigation missions and 456 scenes, each containing a navigational sign and its surrounding environment. We provide synchronized RGB, depth, IMU, and odometry data for each scene. We additionally provide the layout map and the GPS-aligned scene graph for selected mission sites. Beyond navigational decision-making, SiGNgapore supports a variety of spatial-semantic reasoning tasks, including navigational sign parsing and visual sign grounding, with the ultimate aim of driving progress in mapless robot navigation.
Semantic navigation requires an agent to navigate toward a specified target in an unseen environment. Employing an imaginative navigation strategy that predicts future scenes before taking action, can empower the agent to find target faster. Inspired by this idea, we propose SGImagineNav, a novel imaginative navigation framework that leverages symbolic world modeling to proactively build a global environmental representation. SGImagineNav maintains an evolving hierarchical scene graph and uses large vision language models to predict and explore unseen parts of the environment. While existing methods solely relying on past observations, this imaginative scene graph provides richer semantic context, enabling the agent to proactively estimate target locations. Building upon this, SGImagineNav adopts an adaptive navigation strategy that exploits semantic shortcuts when promising and explores unknown areas otherwise to gather additional context. This strategy continuously expands the known environment and accumulates valuable semantic contexts, ultimately guiding the agent toward the target. SGImagineNav is evaluated in both real-world scenarios and simulation benchmarks. SGImagineNav consistently outperforms previous methods, improving the success rate to 65.4% and 66.8% on HM3D and HSSD, and demonstrating cross-floor and cross-room navigation in real-world environments. All source code is open-sourced at https://github.com/UMich-CURLY/SGImagineNav.