OptiSight: Bridging Semantic Reasoning and Geometric Control for Embodied Navigation

Alperen Avan; Jordi Sanchez-Riera

cs.ROAug 24, 2026

OptiSight: Bridging Semantic Reasoning and Geometric Control for Embodied Navigation

Authors: Alperen Avan, Jordi Sanchez-Riera

Organizations: Institut de Rob`otica i Inform`atica Industrial (CSIC-UPC)

Abstract

Autonomous indoor navigation requires both semantic understanding and precise geometric control. We propose OptiSight, a hybrid framework that combines Vision-Language Model reasoning with deterministic visual servoing through a finite-state Chain-of-Thought architecture. Grounded-SAM localizes open-vocabulary targets, while camera projection geometry converts visual observations into navigation commands without requiring dense mapping. The VLM is queried only at key decision points, reducing computational overhead while geometric control handles continuous navigation. Experiments in AI Habitat demonstrate reliable zero-shot navigation across diverse indoor scenarios, including obstacle avoidance and semantic ambiguity, while operating within an 8~GB VRAM budget. The source code is available at https://github.com/avanalperen/OptiSight-Python-Multimodal-CoT-for-Visual-Reasoning.

Figures & tables

Fig. 1 : Conceptual overview of the OptiSight framework. The system perceives the environment, isolates the target via VLM reasoning, and executes a smooth geometric path.

Fig. 2 : Conceptual overview of the OptiSight framework. The system perceives the environment, isolates the target via VLM reasoning, and executes a smooth geometric path.

Fig. 3 : (Step Find) Semantic object localization and segmentation for a given object target.

Fig. 4 : (Step Scan) Obstacle detection given waypoint path to achieve final goal.

Fig. 5 : (Step Navigate) Steering calculation towards the target.

Metric	Exp 01	Exp 02	Exp 03	Exp 04	Exp 05	Exp 06	Exp 07	Exp 08	Exp 09	Exp 10	Exp 11	Exp 12
Mission Success	✓	✓	✓	✗	✗	✗	✓	✗	✓	✗	✓	✓
Success Rate	80%	60%	100%	20%	0%	0%	100%	0%	100%	0%	80%	70%
Execution Time	31.5 s	29.2 s	24.7 s	21.3 s	59.0 s	62.4 s	31.6 s	68.7 s	32.1 s	55.3 s	19.0 s	29.7 s
Total Distance	5.03 m	5.28 m	4.00 m	1.28 m	8.07 m	6.18 m	4.5 m	6.92 m	2.5 m	5.74 m	2 m	6.03 m
Total Steps	64	62	50	35	111	118	50	95	44	71	32	57
Collisions	0	0	0	0	3	6	0	3	0	2	0	0

TABLE I : Performance of OptiSight across Experiments 01–12 using Qwen3.5-2B for VLM reasoning. Results are averaged over 10 independent runs for each scenario.

Panel of Figure 7 — Fig. 6 : Initial frames across Experiments 01–12 indoor navigation scenarios used for the experimental evaluation of OptiSight. Each frame illustrates the initial observation provided to the agent before navigation begins.

Metric	Exp 13	Exp 14	Exp 15	Exp 16	Exp 17	Exp 18	Exp 19	Exp 20	Exp 21	Exp 22	Exp 23	Exp 24
Mission Success	✗	✓	✓	✗	✗	✓	✓	✗	✗	✗	✓	✓
Success Rate	0%	60%	80%	0%	0%	70%	100%	100%	30%	0%	100%	100%
Execution Time	97.3 s	85.4 s	61.0 s	87.6 s	98.9 s	91.8 s	73.1 s	84.0 s	94.2 s	118.1 s	46.5 s	67.7 s
Total Distance	5.59 m	7.96 m	4.75 m	5.37 m	5.12 m	1.49 m	5.0 m	4.5 m	4.5 m	4.48 m	3.75 m	3.49 m
Total Steps	72	88	67	72	77	68	55	70	79	76	40	61
Collisions	2	1	0	2	2	0	0	0	0	4	0	0

TABLE II : Performance of OptiSight across Experiments 13–24 using Moondream2-2B for VLM reasoning. Results are averaged over 10 independent runs for each scenario.

Panel of Figure 9 — Fig. 7 : Initial frames across Experiments 13–24 indoor navigation scenarios used for the experimental evaluation of OptiSight. Each frame illustrates the initial observation provided to the agent before navigation begins.

Robot Navigation

Robot navigation studies how robots perceive, localize, plan, and move safely toward goals in static or changing environments.

See all 557 papers

Bridging the 2D-3D Gap: A Hierarchical Semantic-Geometric Map for Vision Language Navigation
May 25, 2026Kailing Li, Tianwen Qian, Lijin Yang +4VLN Benchmarks 3D Spatial Reasoning
School of Computer Science and Technology, East China Normal University · 2Bosch Corporate Research, Shanghai, China · 3King Abdullah University of Science and Technology
AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation
May 21, 2026Wenxuan Guo, Xiuwei Xu, Yichen Liu +7Vision-Language Navigation VLM Reasoning
Tsinghua University, China
LightNav-0: Eliciting VLM Spatial Intelligence for Generalist Embodied Navigation
Aug 31, 2026Shaoan Wang, Aocheng Luo, Fei Huang +17Open-Vocabulary Object Navigation Cross-Embodiment Robot Learning
Light Origins Team

May 25, 2026J/K move · Enter open · S save

Bridging the 2D-3D Gap: A Hierarchical Semantic-Geometric Map for Vision Language Navigation

Kailing Li, Tianwen Qian, Lijin Yang, Yuqian Fu+3

School of Computer Science and Technology, East China Normal University · 2Bosch Corporate Research, Shanghai, China · 3King Abdullah University of Science and Technology

VLN Benchmarks 3D Spatial Reasoning Vision-Language Navigation Semantic Mapping VLMs for Robotics

Open paper AskPDF arXiv

Vision-Language Navigation (VLN) enables embodied agents to reach target locations in unseen environments by following language instructions. Despite recent progress with vision-language models (VLMs), a critical semantic-geometric gap remains: while VLMs excel at language and 2D visual understanding, they struggle with 3D spatial reasoning and fail to capture the causal dynamics between actions and spatial transitions, resulting in unreliable navigation, particularly in zero-shot settings. To bridge this gap, we propose a Hierarchical Semantic-Geometric Map (HSGM) that transforms 3D geometric information into a structured representation compatible with VLMs, effectively linking them to the physical world. Specifically, HSGM is represented as a multi-channel top-down map organized into three levels: (1) geometric level that records navigable regions and obstacles, (2) semantic level that represents objects and their relations, and (3) decision level that supports high-level task reasoning and goal selection. During navigation, the VLM acts as a high-level semantic planner, interpreting the spatial layout encoded in the HSGM to select geometrically valid waypoints, while low-level, collision-free movements between waypoints are executed by a classical path-planning algorithm, fully decoupling semantic reasoning from action execution. Additionally, complex instructions are decomposed into subtasks to alleviate the problem of progress forgetting or hallucinating in long-horizon navigation. Extensive experiments on R2R-CE and RxR-CE benchmarks demonstrate that our zero-shot framework achieves state-of-the-art performance and even outperforms several supervised methods. Code is available at https://github.com/Teacher-Tom/HSGM_public.

Abstract

Robot Navigation

Explore similar work

Bridging the 2D-3D Gap: A Hierarchical Semantic-Geometric Map for Vision Language Navigation

AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation

LightNav-0: Eliciting VLM Spatial Intelligence for Generalist Embodied Navigation