cs.AIAug 25, 2026
SaveLifePlanner: Evaluating LLM Agents for Geo-spatial Planning with Social Media Data
Organizations: Wuhan University · Hong Kong University of Science and Technology
Abstract
Geo-spatial planning, like trip design, is a realistic testbed for LLM agents because it requires grounded tool use, noisy evidence retrieval, and multi-constraint reasoning. Most benchmarks, however, only provide clean geospatial data and tools, missing the open-ended social signals that people use in daily planning. We introduce LifePlanner, a benchmark that enriches map data with large-scale local social media posts and provides access through an MCP toolset. LifePlanner provides an evaluation suite spanning four task categories and three difficulty levels. Experiments show frontier LLMs perform well on simple retrieval but degrade sharply on complex planning, with the Pass Rate dropping to 40.2%. Results show that failures mainly stem from incomplete evidence acquisition from such a large multimodal database, imprecise tool use, and weak constraint integration rather than model size or reasoning length, suggesting that future progress requires effective grounded planning instead of scaling alone.
Figures & tables
| Work | Env. | Eval. | ||
| Tool Box | Social Media | Multi Task | Multi Diff. | |
| TravelPlanner Xie et al. (2024) | ✓ | ✗ | ✗ | ✓ |
| CityEval Feng et al. (2025a) | ✗ | ✗ | ✓ | ✗ |
| USTBench Lai et al. (2025) | ✗ | ✗ | ✓ | ✗ |
| CityBench Feng et al. (2025c) | ✗ | ✗ | ✓ | ✗ |
| MapEval Dihan et al. (2025) | ✓ | ✗ | ✓ | ✗ |
Table 1: Comparison with prior geo-spatial benchmarks. Multi Task indicates the inclusion of varying geo-spatial task types, and Multi Diff. denotes the presence of hierarchical task difficulty levels.
Figure 1: LifePlanner Overview. LifePlanner consists of two main components: an environment and an evaluation protocol. The environment provides geospatial and social-media evidence through an MCP server, while the evaluation assesses both the tool-use process and final outputs of LLM agents.
Figure 2: LifePlanner uses social media data to enrich map data and construct a rich, realistic environment. Note that evaluated LLMs cannot access the time-sensitive information (mem field) extracted from social media.
Figure 3: LifePlanner Task System. The benchmark spans four task categories and three difficulty levels. L0 requires a single database retrieval step (green), L1 requires multiple retrieval steps and unconstrained calculation, and L2 further requires implicit constraints (red) to be integrated with retrieved evidence for a globally valid plan.
| Model | Place Perception | Nearby Discovery | Routing | Trip | Overall | ||||||||
| L0 | L1 | L2 | L0 | L1 | L2 | L0 | L1 | L2 | L1 | L2 | Avg. | #Tokens | |
| Closed-source Models | |||||||||||||
| Claude-Sonnet-4.6 | 99.0 | 87.5 | 91.4 | 74.0 | 67.3 | 54.3 | 100.0 | 46.0 | 38.7 | 69.8 | 25.7 | 62.7 | 121.8k |
| Qwen3.6-Plus | 99.0 | 84.7 | 88.6 | 87.5 | 61.5 | 48.6 | 98.0 | 44.0 | 35.5 | 69.8 | 25.7 | 63.3 | 142.3k |
| GPT-5.4 | 97.0 | 83.3 | 80.0 | 75.0 | 44.2 | 40.0 | 95.0 | 44.0 | 29.0 | 66.0 | 28.6 | 57.9 | 98.9k |
| Gemini-3.1-Pro-Preview | 98.0 | 86.1 | 71.4 | 78.9 | 53.9 | 48.6 | 100.0 | 20.0 | 16.1 | 41.5 | 14.3 | 59.4 | 56.5k |
Table 2: [email protected] in % (Pass Rate with ) of LLMs. We mark the best result in bold and the second-best with underlining .
| Metric | L0 | L1 | L2 |
| Process | |||
| Information Grounding | 96.5 | 80.4 | 64.6 |
| Tool Efficiency | 76.2 | 53.7 | 44.6 |
| Reasoning Purity | 96.7 | 89.4 | 86.4 |
| Outcome | |||
| Attribute Accuracy | 96.1 | 86.9 | 80.8 |
Table 3: Average performance in % of all LLMs across different task difficulty levels. “-” indicates that Order Consistency and Semantic Constraint are not applicable to the single-hop retrieval nature of L0 tasks.
Figure 4: Average scores (left) and strict Pass Rates (right) of representative models across the three difficulty levels (L0, L1, L2).
Figure 5: Cross-model exceedance curves for Reasoning Purity, Information Grounding, and Tool Efficiency across different L2 task types. A point indicates that of cases achieved a score , where is expressed as a percentage.
Figure 6: Distribution of tool-call failures caused by parameter errors, truncation due to overly broad conditions, and empty results or errors due to imprecise conditions.
Figure 7: Representative failure cases. Yellow highlights indicate the main errors: (Left) the model repeatedly retrieves the wrong target with an incorrect tag until the tool budget is exhausted; (Right) the model fails to jointly consider constraints such as store closing times when composing the final plan.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Task Category | L0 | L1 | L2 | Total |
| Place Perception | 100 | 72 | 35 | 207 |
| Nearby Discovery | 104 | 52 | 35 | 191 |
| Routing | 100 | 50 | 31 | 181 |
| Trip Planning | – | 53 | 35 | 88 |
| Total | 304 | 227 | 136 | 667 |
Table 4: Distribution of tasks across different categories and difficulty levels. Note that L0 is not applicable to Trip tasks, as they inherently require multi-location retrieval and navigation.
| Parent Field | Child Field | Description |
| type | – | Flag indicating task subtype and difficulty level. (e.g., routing_L1, trip_L2) |
| question | – | Natural-language user query presented to the benchmark agent. |
| answer | ( Self ) | Root payload object or array containing the final task results or scalar answer. |
| name | Name of a returned place, start/end point, waypoint, or itinerary stop. | |
| place_id | Map identifier corresponding to the returned location. | |
| reason | Natural-language explanation for a verification, comparison, route selection, or stop choice. |
Table 5: Field definitions for query instances across different tasks. Nested fields are logically grouped under their parent objects.
| Task Family | Level | Included Fields |
| Nearby | L0 | type , question , answer , name , place_id , reason , ref , attribute_key |
| L1 | type , question , answer , name , place_id , reason , ref , attribute_key , note_id | |
| L2 | type , question , answer , name , place_id , reason , ref , attribute_key , note_id , implicit_logic | |
| Place | L0 | type , question , answer , ref , attribute_key , note_id |
| L1 | type , question , answer , reason , ref , attribute_key , note_id | |
| L2 | type , question , answer , reason , ref , attribute_key , note_id , implicit_logic |
Table 6: Field coverage by task family and difficulty level. Raw instance fields have been mapped to the simplified ontology defined in Table 5 . Note that Trip lacks an L0 configuration due to its inherent complexity.
| Tool Name | Description | Parameters |
| Map Domain | ||
| place_search | Search places by name with lightweight POI output ( place_id , name , type , rating , cost ). Supports type, rating, cost, and geo-tag filtering. | query_name : REQUIRED: Place name in local language place_types : POI types list min_rating / max_rating : Min/Max rating (1-5) min_cost / max_cost : Min/Max cost per capita (CNY) geo_tag : Filter by tag/name keywords list in local language, matches if ANY keyword hits |
| place_detail | Get full POI information for one place, including name, type, address, hours, rating, cost, tags, and alias. | place_id : POI place_id (from place_search or nearby_search) |
| nearby_search | Search nearby places around a POI or along a route buffer. Standard agent version returns a lightweight POI list with distance. | center : Center: POI place_id (str) OR Route object {’start’: ’ID’, ’end’: ’ID’, ’vehicle’: ’driving’} radius_meters : Search radius/buffer in meters (default 100) place_types : POI types list min_rating / max_rating : Min/Max rating (1-5) min_cost / max_cost : Min/Max cost per capita (CNY) geo_tag : Filter by tag/name keywords list in local language, matches if ANY keyword hits |
| routing | Calculate an optimal route between two POIs and return total distance/time, geometry, and turn instructions. | start_place_id : Start POI place_id end_place_id : End POI place_id avoid_way_ids : Way IDs to avoid (ways.gid) mode : Route mode: ’time’ (shortest time) or ’distance’ (shortest distance) vehicle : Travel mode: ’walking’, ’cycling’, or ’driving’ |
| submit_answer | Submit the final structured JSON answer. The evaluated agent must call this last instead of printing plain JSON. | result : Your final answer as a JSON object or array following the required format |
Table 7: The Standard Agent Toolset. Tools are designed to be lightweight, requiring the evaluated agent to execute multi-step reasoning and drill-down tool invocations to fulfill complex queries.
| Tool Name | Description | Parameters |
| Map Domain | ||
| place_search | Search places by name with enhanced filters and return full POI records. Data Engine version can filter by both static geo tags and social-memory keywords. | query_name : REQUIRED: Place name in local language place_types : POI types list, separated by commas (,) min_rating / max_rating : Min/Max rating (1-5) min_cost / max_cost : Min/Max cost per capita geo_tag : Filter by tag/name keywords list social_mem : Filter by social insights in local language |
| nearby_search | Search nearby places around a POI or along a route buffer and return full POI records, including address , opentime_week , tag , alias , long_mem , and short_mem . | center : Center: POI place_id (str) OR Route object {’start’: ’ID’, ’end’: ’ID’, ’vehicle’: ’driving’} radius_meters : Search radius/buffer in meters place_types : POI types list, separated by commas min_rating / max_rating : Min/Max rating (1-5) min_cost / max_cost : Min/Max cost per capita geo_tag : Filter by tag/name keywords list social_mem : Filter by long_mem/short_mem keywords (social media insights) in local language |
| routing | Calculate an optimal route between two POIs and return total distance/time, geometry, and turn instructions. | start_place_id : Start POI place_id end_place_id : End POI place_id avoid_way_ids : Way IDs to avoid (ways.gid) mode : Route mode: ’time’ or ’distance’ vehicle : Travel mode: ’walking’, ’cycling’, or ’driving’ |
| sample_place | Randomly sample POIs from the map database for prompt initialization. Returns full POI information including memory fields. | num_samples : Number of random POIs to sample (typically 1-50) |
| multi_stop_routing | Enumerate and score route permutations for multi-stop trip planning. Supports open trips and fixed start/end settings. | stop_ids : List of POI place_ids to visit (typically 2-6 stops) start_id : Fixed start place_id (must be in stop_ids) end_id : Fixed end place_id (must be in stop_ids) vehicle : Travel mode: ’walking’, ’cycling’, ’driving’ mode : Optimize: ’time’ or ’distance’ |
Table 8: The Data Engine Agent Toolset. Enhanced MCP tools directly return highly dense and comprehensive records to ensure complete information retrieval during dataset generation without information loss.
| Returned Key | Exposed by Tool(s) | Explanation |
| osm_id | place_search , nearby_search , sample_place , place_detail | Primary POI identifier. |
| name | place_search , nearby_search , sample_place , place_detail | POI name. |
| place_type | place_search , nearby_search , sample_place , place_detail | Categorization of the POI (e.g., Auto Service|Dedicated Charging Station). |
| address | place_search (DE), nearby_search (DE), sample_place , place_detail | Physical address of the POI. |
| opentime_week | place_search (DE), nearby_search (DE), sample_place , place_detail | Weekly business opening hours, if available. |
| rating | place_search , nearby_search , sample_place , place_detail | User rating score. |
Table 9: Retrievable fields from the Map Database. Fields marked with (DE) are exclusively exposed upfront to the Data Engine Agent during the initial search.
| Returned Key | Exposed by Tool(s) | Explanation |
| note_id | search_notes , note_detail | Primary identifier for the social media post. |
| title | search_notes , note_detail | Title of the post. |
| desc | search_notes (DE), note_detail | Full description and textual content of the post. |
| time | search_notes , note_detail | Publication timestamp of the post. |
| liked_count | search_notes , note_detail | Number of likes received by the post. |
| collected_count | search_notes , note_detail | Number of times the post has been saved or favorited. |
Table 10: Retrievable fields from the Social Media Database. The Data Engine Agent (DE) retrieves detailed comment data directly from the search tool to ensure data integrity during generation.
| Metric | Place Perception | Nearby Discovery | Routing | Trip Design | |||||||
| L0 | L1 | L2 | L0 | L1 | L2 | L0 | L1 | L2 | L1 | L2 | |
| Attr Accuracy | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |||
| Place-set IoU | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |||
| Routing Deviation | ✓ | ✓ | ✓ | ✓ | ✓ | ||||||
| Order Consistency | ✓ | ✓ | |||||||||
| Semantic Constraint | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |||
Table 11: Applicability matrix of evaluation metrics across different task types and difficulty levels.