Reliable local search is essential for LLM agents to support everyday decisions and longer plans by finding places that meet users' needs at given times and locations. Yet existing benchmarks provide limited coverage of whether search results satisfy mandatory requirements, account for preferences, and ground factual and experiential judgments in place attributes and reviews. We introduce ATLAS-Bench, a benchmark of 1,200 queries across three regions, expanded from real search events using place attributes, reviews, and existing user profiles from the platform. Each query distinguishes mandatory requirements from preferences and factual conditions from subjective judgments, with supporting evidence and criteria for checking each requirement. Queries are validated through iterative assertion checks and cross-model review. We also introduce ATLASBox, a standardized evaluation environment with search tools and a fixed point-of-interest (POI) knowledge base. Given a query, timestamp, and location, an agent searches place attributes and reviews to recommend evidence-supported options or report that no place meets all mandatory requirements. We evaluate answer quality, trajectory quality, and execution cost. Across 14 configurations of 11 models, GPT-6-Luna achieves the highest constraint satisfaction rate among returned recommendations (71.5%) and factual accuracy (95.6%). However, it returns at least one acceptable recommendation for fewer satisfiable queries than GLM-5.3 (67.3% versus 85.3%). These results highlight the challenge of satisfying individual requirements while providing usable recommendations across queries.
Figures & tables
Figure 1. Overview of ATLAS-Bench. (a) Query-only search lacks context and treats all constraints alike. (b) ATLAS-Bench adds context, constraint types, and evidence. (c) Local search within a longer plan. An overview comparing traditional search and agentic local search with multi-faceted constraints.
Dimension
MapEval
LocalSearch
Mobility
MapSatisfy
ATLAS
Query source
Manual
Manual
Trajectory
Map logs
Log-grounded
Reference target
Manual
Manual
Destination
Decision rubrics
Clicked POI
Context
✗
△
✓
✓
✓
Constraint taxonomy
✗
△
✗
△
✓
Constraint conflicts
✗
✗
✗
✗
✓
Implicit needs
✗
△
✗
✓
✓
Table 1. Comparison of local search and mobility benchmarks. ✓ : supported; △ : partial or limited; ✗ : not supported.
Figure 2. Four constraint combinations for a shared restaurant-search intent. A two-by-two matrix illustrating subjective and objective constraints, each expressed as soft preferences or hard requirements.
Figure 3. Overview of ATLAS. (a) Search logs are linked to POI attributes and reviews, privacy-rewritten and validated, and stored as vectors and raw records. (b) Persona-aware queries pass through assertions, cross-model debate, repair, and finalization; failures update the assertion memory from A(t) to A(t+1) . (c) Agents solve the tasks in ATLASBox and are evaluated on answer quality, trajectory quality, and efficiency.
Metric
Chaoyang
Huangpu
Dabancheng
(a) Corpus statistics
Queries
400
400
400
Active POIs
343,916
79,058
2,270
Mean constraints/query
2.17
3.09
2.37
Intent entropy
0.8329
0.6264
0.6750
Constraint combination entropy
0.6608
0.8130
0.5788
Table 2. Regional statistics and constraint coverage. A query may contain multiple constraint types.
Figure 4. Observed synthesis-quality trajectories across ten RSI rounds.
Answer Quality
Trajectory Quality
Efficiency
Model
CSR
CR
Success@k
Recs/q
Factual
Intent
Approp.
Evidence
Strategy
Tokens/q
Calls/q
Rounds/q
Qwen3.5-4B
w/o Think
37.3
65.8
54.0
2.24
40.25
3.53
3.70
1.99
2.96
36.8K
3.79
1.92
w/ Think
44.0
71.4
57.7
1.81
54.33
3.17
3.84
2.11
3.19
50.3K
4.74
2.73
Qwen3.6-35B-A3B
w/o Think
46.5
70.9
73.1
2.43
49.83
3.93
3.99
2.27
3.50
57.6K
6.07
3.33
Table 3. Overall performance on ATLAS-Bench. The best and second-best values are highlighted, excluding efficiency.
Figure 5. Correlations between trajectory and answer quality.
Figure 6. POI distributions, colored by area-normalized density on a shared log scale.
Answer Quality
Trajectory
Efficiency
Region
CSR
CR
Success@k
Recs (/query)
Factual accuracy
Evidence sufficiency
Tool calls
Tokens
Chaoyang
62.9
77.1
80.0
2.81
72.6
2.46
6.04
70.5K
Huangpu
41.4
68.8
63.7
2.08
68.6
1.90
6.12
75.4K
Dabancheng
68.7
84.0
80.8
2.22
81.2
2.91
3.88
43.0K
Table 4. Regional performance across 14 configurations.
C=100
C=200
N
Metric
T=10
T=20
T=30
T=10
T=20
T=30
2
3
4
5
6
7
8
CSR (%)
56.4
60.4
56.4
59.0
59.3
53.3
55.9
52.8
55.8
55.0
58.3
59.1
56.8
CR (%)
75.5
74.4
74.0
76.7
77.5
73.6
77.2
73.5
75.5
74.0
74.1
76.1
73.6
Success@k (%)
74.4
77.9
82.6
76.7
79.1
76.7
73.3
75.6
79.1
80.2
80.2
81.4
77.9
Recs/q
2.24
2.47
2.73
2.34
2.48
2.71
2.45
2.44
2.74
2.52
2.56
2.72
2.60
Factual (%)
81.00
76.00
72.00
80.00
72.00
77.00
79.00
73.00
79.00
77.00
76.00
78.00
76.00
Table 5. Budget sensitivity of Qwen3.8-27B (w/o thinking). C : candidate-pool size; T : results per call; N : maximum rounds.
Method
k
NDCG@10
Hit@10
Recall@20
MRR@20
P50 (s)
BM25
–
0.606
0.813
0.632
0.689
0.059
Vector
100
0.664
0.881
0.675
0.771
0.057
BM25 + rerank
100
0.726
0.884
0.713
0.820
0.564
Vector + rerank
20
0.716
0.899
0.675
0.817
0.160
Vector + rerank
100
0.745
0.927
0.745
0.833
0.568
Vector + rerank
200
0.747
0.935
0.757
0.831
1.078
Table 6. Retrieval comparison across candidate-pool sizes k . Shading marks the default; bold indicates the best value.
Figure 7. Recommendation richness: (a) list-size distributions and (b) quantity versus conditional intra-list diversity.
Figure 8. Contrasting tool-use trajectories: (a) evidence-guided candidate refinement in a high-scoring case; (b) repetitive, off-target retrieval in a failure case.
Figure 9. Long-horizon case with stage choices and evidence.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10. Spatial distributions of the eight most frequent POI categories in Chaoyang, Huangpu, and Dabancheng. Each panel reports a category’s POI count and share of the regional collection. Maps of the leading eight POI categories in each of three benchmark regions, with counts, percentages, and a shared logarithmic density legend.
Figure 11. Condensed prompts for Generate, Debate, and Fix, showing instructions, runtime inputs, and expected outputs.
Figure 12. ATLAS-Bench composition. (a) Shares of 3,049 atomic constraint annotations by dimension. (b) Shares of 1,200 queries by task type, with S1–S8 aggregated. Two donut charts showing 20 constraint dimensions and six task groups. The largest constraint dimensions are location, category, and UGC reputation. Single-constraint queries total 300; M1 through M5 contain 196, 196, 197, 196, and 115 queries.
Benchmark View
Semantic Dist ↑
Surface Div ↑
ATLAS-Bench Pooled
0.5844
0.4622
ATLAS-Bench Beijing
0.5898
0.4067
ATLAS-Bench Shanghai
0.5630
0.4327
ATLAS-Bench Urumqi
0.5338
0.2854
LocalSearchBench Question
0.4946
0.0629
LocalSearchBench Hop
0.5830
0.1697
Appendix
Table 7. Comparison of semantic and surface diversity at 400 instances.
Check
Observed issue
Revision
Name leakage
Geographic anchors falsely flagged.
Exempt geographic terms; fix whitespace matching.
Evidence linkage
POI-based joins mixed search events.
Join evidence by source-record ID.
Semantic rejection
Lexical rules rejected valid queries.
Broaden matching; make the check advisory.
Conflict references
Mixed indexing linked wrong constraints.
Normalize index conventions.
Constraint types
Overlapping fields conflated dimensions.
Refine matching; separate semantic dimensions.
Satisfiability
Default labels masked conflicts.
Remove positive defaults; repair conflict logic.
Appendix
Table 8. Representative assertion and validation revisions.
Tool
Purpose
Inputs
search_pois
Retrieve POIs and structured attributes; resolve landmark coordinates.
query
search_notes
Retrieve review evidence for experiential requirements.
query
get_poi_notes
Inspect reviews linked to a specific POI.
poiid
search_nearby
Find POIs within a radius of the given coordinates.
query, loc, distance_m
Appendix
Table 9. ATLASBox search tools and required inputs. Optional arguments control result limits and filters.