Route-planning agents powered by large language models (LLMs) have emerged as a promising paradigm for supporting everyday human mobility through natural language interaction and tool-mediated decision making. However, systematic evaluation in real-world mobility settings is hindered by diverse routing demands, non-deterministic mapping services, and limited reproducibility. In this study, we introduce MobilityBench, a scalable benchmark for evaluating LLM-based route-planning agents in real-world mobility scenarios. MobilityBench is constructed from large-scale, anonymized real user queries collected from Amap and covers a broad spectrum of route-planning intents across multiple cities worldwide. To enable reproducible, end-to-end evaluation, we design a deterministic API-replay sandbox that eliminates environmental variance from live services. We further propose a multi-dimensional evaluation protocol centered on outcome validity, complemented by assessments of instruction understanding, planning, tool use, and efficiency. Using MobilityBench, we evaluate multiple LLM-based route-planning agents across diverse real-world mobility scenarios and provide an in-depth analysis of their behaviors and performance. Our findings reveal that current models perform competently on Basic information retrieval and Route Planning tasks, yet struggle considerably with Preference-Constrained Route Planning, underscoring significant room for improvement in personalized mobility applications. We publicly release the benchmark data, evaluation toolkit, and documentation at https://github.com/AMAP-ML/MobilityBench.
Figures & tables
Figure 1. Overview of MobilityBench, a systematic benchmark for evaluating route-planning agents.
Intent Family
Task Scenario
Example Query
Basic Information Retrieval
POI Query
Where is the gas station?
Geolocation Query
Where am I?
Nearby Query
Search for restaurants near Beijing Capital International Airport.
Weather Query
What is the weather like in Wuhan tomorrow?
Traffic Info Query
Is there a traffic jam on Chengdu Avenue right now?
Route-Dependent Information Retrieval
Route Property Query
How far is it from Hefei to Huangshan?
Table 1. Overview of task scenarios in MobilityBench, grouped by intent family.
Figure 2. Global coverage of MobilityBench Data.
Table 2. Performance of models on MobilityBench. Abbreviations: Instr. Und. for Instruction Understanding; Dec. Mak. for Decision Making; ID for Intent Detection; IE for Information Extraction; DEC for Task Decomposition; TS for Tool Selection; SC for Schema Compliance; DR for Delivery Rate; FPR for Final Pass Rate; IT for Input Token; and OT for Output Token.