Vision-Language Navigation (VLN) requires an embodied agent to navigate complex environments by following natural language instructions, which typically demands tight fusion of visual and language modalities. Existing VLN methods often convert raw images into visual tokens or implicit features, requiring large-scale visual pre-training and suffering from poor generalization under environmental variations (e.g., lighting, texture). To address these issues, we propose SOL-Nav (Structured Observation Language for Navigation), a novel framework that translates egocentric visual observations into compact structured language descriptions for efficient and generalizable navigation. Specifically, we divide RGB-D images into a NxN grid, extract representative semantic, color, and depth information for each grid cell to form structured text, and concatenate this with the language instruction as pure language input to a pre-trained language model (PLM). Experimental results on standard VLN benchmarks (R2R, RxR) and real-world deployments demonstrate that SOL-Nav significantly reduces the model size and training data dependency, fully leverages the reasoning and representation capabilities of PLMs, and achieves strong generalization to unseen environments.
Figures & tables
Figure 1: Pipeline of SOL-Nav. RGB-D observations are converted into structured textual descriptions with 2×2/4×4/6×6 multi-resolution grids (long/short-term history, current observation) encoding depth, semantic, and color information. The structured observation sequence, navigation instruction, and system description form a pure language prompt, which is input to a LLM to predict a consecutive action block for the agent.
Figure 2: Structured Observation Language Prompt for LLM. The prompt integrates system description ( Dsystem ), structured observation ( Ostructure ), and task instruction ( Itask ) to provide clear system definition, structured observations, and explicit prediction requirements for the language model.
Method
Observation
R2R Val-Unseen
Pano.
Odo.
Depth
S.RGB
NE ↓
OS ↑
SR ↑
SPL ↑
HPN+DN* ( Krantz et al. 2021 )
✓
✓
✓
6.31
40.0
36.0
34.0
CMA* ( Hong et al. 2022 )
✓
✓
✓
6.20
52.0
41.0
36.0
Sim2Sim* ( Krantz and Lee 2022 )
✓
✓
✓
6.07
52.0
43.0
36.0
GridMM* ( Wang et al. 2023a )
✓
✓
✓
5.11
61.0
49.0
41.0
ScaleVLN* ( Wang et al. 2023b )
✓
✓
✓
4.80
–
55.0
51.0
Table 1: Comparison with state-of-the-art methods on VLN-CE R2R Val-Unseen split. ∗ indicates methods using the waypoint predictor from ( Hong et al. 2022 ) . † denotes methods using additional training data beyond the R2R-CE benchmarks.
Method
Observation
RxR Val-Unseen
Pano.
Odo.
Depth
S.RGB
NE ↓
OS ↑
SR ↑
SPL ↑
CMA ‡ * ( Hong et al. 2022 )
✓
✓
✓
11.7
10.7
4.4
2.5
ETPNav ‡ * ( An et al. 2024 )
✓
✓
✓
8.79
36.7
25.5
18.1
CM2 ‡ ( Georgakis et al. 2022b )
✓
✓
✓
8.98
25.3
14.4
9.2
LAW ‡ ( Raychaudhuri et al. 2021 )
✓
✓
✓
10.87
21.0
8.0
8.0
WS-MGMap ‡ ( Chen et al. 2022a )
✓
✓
✓
9.83
29.8
15.0
12.1
Table 2: Comparison with state-of-the-art methods on the VLN-CE RxR Val-Unseen splits. ‡ indicates methods that only trained on VLN-CE R2R. ∗ indicates methods that utilize the waypoint predictor proposed by ( Hong et al. 2022 ) . † denotes methods that employ additional training data beyond the RxR-CE benchmarks.
Ablation Term
Observation Config
Metrics
Resolution
History
Depth
Semantic
Color
NE ↓
OS ↑
SR ↑
SPL ↑
Lower Res.
4×4
✓
✓
✓
✓
6.84
43.4
34.5
29.8
No His.
6×6
✓
✓
✓
7.81
39.4
26.5
21.9
No Depth
6×6
✓
✓
✓
7.98
31.2
21.6
17.8
No Semantic
6×6
✓
✓
✓
8.30
28.1
20.3
15.2
No Color
6×6
✓
✓
✓
7.45
63.8
48.7
41.5
Table 3: Ablation study results on the R2R-CE val-unseen split, evaluating the impact of five core components of SOL-Nav: grid resolution (6×6 vs. 4×4), historical observation, depth, semantic segmentation, and color information. The full model (6×6 resolution, with all five inputs) serves as the baseline for all ablation variants.
Method
Total Latency (s) ↓
GPU Memory (GB) ↓
NaVILA
0.595
18.5
InternVLA-N1
0.730
20.0
Ours
0.091
2.2
Table 4: Computational cost comparison results.
Device
Segmentation
SOP
Qwen3-E
Total
Jetson Orin
0.032
0.220
0.533
0.785
RTX 4090
0.003
0.023
0.065
0.091
Table 5: Inference latency (s) details on different devices.
Figure 3: Real-world Deployments. We conduct real-world navigation experiments in three distinct scenarios with varying environmental characteristics (Tea Area, Hall Stairs, Meeting Room), to comprehensively evaluate the robustness and generalization of SOL-Nav.