Time-Aware Assistive Navigation

Masaki Kuribayashi; Zhongkai Shangguan; Eshed Ohn-Bar

cs.CVSep 4, 2026

Time-Aware Assistive Navigation

Authors: Masaki Kuribayashi, Zhongkai Shangguan, Eshed Ohn-Bar

Organizations: Waseda University · Boston University

Abstract

Can interactive vision-and-language agents learn not just what to say but also when to say it? Current language models rarely plan over whether and when to realize a real-time response to a user. However, providing accurate and timely support for human decision-making, such as when guiding visually impaired individuals through urban environments, requires careful real-time responsiveness--poorly timed responses can distract users or add unnecessary cognitive load. As a machine intelligence challenge for Multimodal Large Language Model (MLLM)-based agents, we introduce a large-scale multimodal benchmark for an egocentric, assistive navigation task in complex outdoor environments. Using this benchmark, we uncover a fundamental limitation of off-the-shelf MLLMs in delivering safe and time-sensitive navigation instructions, even with model fine-tuning on substantial amounts of data. We then demonstrate that a simple yet effective modification of the model, including direct supervision to predict the underlying reason for each instruction, yields significant performance gains across open-loop, closed-loop, and sim-to-real generalization settings. However, our analysis highlights persistent challenges in temporal reasoning, safety-critical object awareness, and relational and distance understanding. To advance the development of scalable assistive agents, we will release our simulation, benchmark, and code (available at the project website: https://timeli-icra.github.io/).

Figures & tables

Fig. 1 : Step-by-Step Video Instruction Generation Task. We introduce TIMELI , a novel machine intelligence task and model for providing temporally-aware, concise, and safe assistance to blind users navigating toward a goal. Based on our analysis, we demonstrate how current MLLMs, e.g. , ChatGPT (depicted in the figure), provide ineffective and overly verbose instructions for our task. In contrast, the TIMELI-based agent reasons about both what to say and when to say it, ensuring alignment with the needs of blind users, e.g. , remaining silent at intersections to reduce cognitive load and prevent hazardous situations, which is a standard strategy employed by mobility guides.

Dataset	Dynamic	Context	Temporal Information	For Blind End-User	Size	Collection
R2R [ 62 ]	✗	Indoor	✗	✗	21,567 Images	Real-World
CVDN [ 70 ]	✗	Indoor	✗	✗	7,000 Images	Real-World
REVERIE [ 71 ]	✗	Indoor	✗	✗	21,702 Images	Real-World
Touchdown [ 64 ]	✓	Outdoor	✗	✗	9326 Images	Real-World
Talk the Walk [ 72 ]	✓	Outdoor	✗	✗	10,000 Images	Real-World
RxR [ 51 ]	✗	Indoor	✗	✗	126,069 Images	Real-World

TABLE I : Comparing Vision-and-Language Navigation Datasets. Compared to prior datasets, our benchmark includes rich contextual instructions and temporal information, specifying when and what to say to guide a visually impaired user. Additionally, we emphasize the involvement of dynamic obstacles ( e.g. , pedestrians) in outdoor navigation scenarios. For comparison with other non-temporal benchmarks, we also note the size of each dataset (total number of images in TIMELI is

1{,}233{,}616

). While the synthetic benchmark is procedurally generated to incorporate diverse factors (e.g., weather, time of day, safety hazard), we further curate a real-world dataset sourced from an in-situ study and manually annotated YouTube videos, reserved for validation.

Fig. 2 : Model Structure and Task Overview. The model takes input video, plan (defined from a set of map-based waypoints), and task specifications with historical instructions (Sec. III-A ). We explicitly supervise for instructional reason (Sec III-C ), which we find improves model guidance performance.

Model	Size	Modality	BLEU-4	ROUGE-L	Timing F1	Timing AUC	Conciseness
Off-the-Shelf Models
VILA [ 25 ]	3B	Image	0.626	6.545	0.579	0.778	0.554
TinyLLaVA [ 107 ]	3B	Image	0.000	5.728	0.361	0.500	0.264
VILA [ 25 ]	3B	Video	0.000	4.377	0.465	0.651	0.552
LLaVA-v1.6 [ 108 ]	7B	Image	1.849	7.851	0.575	0.776	0.068
NaVid [ 60 ]	7B	Video	0.000	6.659	0.384	0.606	0.046

TABLE II : Open-Loop Assistive Instruction Generation Using Off-the-Shelf Models and Finetuned Models. We sequentially provide models with images and videos in the TIMELI benchmark to evaluate their safety-critical navigation performance. The output of the off-the-shelf models is guided via in-context learning. While all models have access to goal coordinates, the finetuned model denoted by “+” incorporates instruction history, high-level plan, and reason, while those denoted by “++” additionally include timing supervision.

Model	Size	Modality	BLEU-4	ROUGE-L	Timing F1	Timing AUC	Conciseness
TinyLLaVA+	3B	Image	0.065	1.592	0.454	0.500	0.019
VILA+	3B	Video	7.005	20.359	0.186	0.545	0.253
TimeChat+	7B	Video	0.001	22.995	0.025	0.506	0.383
LLaVA-v1.6+	7B	Image	8.478	22.587	0.732	0.815	0.278
LLaVA-v1.6++	7B	Image	9.479	23.970	0.672	0.771	0.266

TABLE III : Zero-Shot Transfer from Simulation to Real-World. We evaluate models trained on the TIMELI benchmark in open-loop settings on real-world videos. We find models transfer knowledge from our extensive simulation benchmark to real-world scenarios without any further fine-tuning.

Model	Size	Modality	NQS	Success Rate	Route Completion	Navigation Score	Collision/Min	Instructions/Min
TinyLLaVA	3B	Image	0.00	0.00	0.01	0.01	0.00	0.00
TinyLLaVA+	3B	Image	0.81	0.47	0.93	0.92	0.51	18.91
VILA	3B	Video	0.29	0.02	0.55	0.44	0.68	8.66
VILA+	3B	Video	0.57	0.14	0.79	0.63	0.60	16.35
TimeChat	7B	Video	0.24	0.04	0.45	0.46	0.41	2.95
TimeChat+	7B	Video	0.38	0.17	0.68	0.54	0.51	3.66

TABLE IV : Closed-Loop Performance Using Finetuned Models. We perform a study where pedestrians are controlled based on instructions generated from models to walk along 12 routes in the simulation environment (see metric definition in Sec. IV-A ). Note that some models may achieve low errors due to minimal progress along the route.

Visually Impaired Navigation Assistance

See all 22 papers

Can MLLMs Find Their Way in a City? Exploring Emergent Navigation from Web-Scale Knowledge
Dec 17, 2025Dwip Dalal, Utkarsh Mishra, Narendra Ahuja +1Visual Navigation Vision-Language Navigation
University of Illinois Urbana-Champaign · Texas A&M University · Microsoft Research, Redmond
NavHarness: Adaptive Goals for Agentic Vision-Language Navigation
Sep 30, 2026Haoxiang Shi, Zaijing Li, Muhe Ding +3Vision-Language Navigation Multimodal Agents
Harbin Institute of Technology (Shenzhen) · Pengcheng Laboratory
Urban Risk-Aware Navigation via VQA-Based Event Maps for People with Low Vision
May 12, 2026Antoni Valls, Jordi Sanchez-RieraVisual Question Answering Vision-Language Models
Institut de Rob`otica i Inform`atica Industrial, CSIC-UPC, Llorens i Artigas 4-6, 08028 Barcelona, Spain

Dec 17, 2025J/K move · Enter open · S save

Can MLLMs Find Their Way in a City? Exploring Emergent Navigation from Web-Scale Knowledge

Dwip Dalal, Utkarsh Mishra, Narendra Ahuja, Nebojsa Jojic

University of Illinois Urbana-Champaign · Texas A&M University · Microsoft Research, Redmond

Visual Navigation Vision-Language Navigation Multimodal Spatial Reasoning Spatial Reasoning

Open paper AskPDF arXiv

Leveraging multimodal large language models (MLLMs) to develop embodied agents offers significant promise for addressing complex real-world tasks. However, current evaluation benchmarks remain predominantly language-centric or heavily reliant on simulated environments, rarely probing the nuanced, knowledge-intensive reasoning essential for practical, real-world scenarios. To bridge this critical gap, we introduce the task of Sparsely Grounded Visual Navigation, explicitly designed to evaluate the sequential decision-making abilities of MLLMs in challenging, knowledge-intensive real-world environment. We operationalize this task with CityNav, a comprehensive benchmark encompassing four diverse global cities, specifically constructed to assess raw MLLM-driven agents in city navigation. Agents are required to rely solely on visual inputs and internal multimodal reasoning to sequentially navigate 50+ decision points without additional environmental annotations or specialized architectural modifications. Crucially, agents must autonomously achieve localization through interpreting city-specific cues and recognizing landmarks, perform spatial reasoning, and strategically plan and execute routes to their destinations. Through extensive evaluations, we demonstrate that current state-of-the-art MLLMs, reasoning techniques (e.g., GEPA, chain-of-thought, reflection) and competitive baseline PReP significantly underperform in this challenging setting. To address this, we propose Verbalization of Path(VoP), which explicitly grounds the agent's internal reasoning by probing city-scale cognitive maps (key landmarks and directions toward the destination) from the MLLM, substantially enhancing navigation success. Project Webpage: https://dwipddalal.github.io/AgentNav/

Abstract

Visually Impaired Navigation Assistance

Explore similar work

Can MLLMs Find Their Way in a City? Exploring Emergent Navigation from Web-Scale Knowledge

NavHarness: Adaptive Goals for Agentic Vision-Language Navigation

Urban Risk-Aware Navigation via VQA-Based Event Maps for People with Low Vision