Reliable local search is essential for LLM agents to support everyday decisions and longer plans by finding places that meet users' needs at given times and locations. Yet existing benchmarks provide limited coverage of whether search results satisfy mandatory requirements, account for preferences, and ground factual and experiential judgments in place attributes and reviews. We introduce ATLAS-Bench, a benchmark of 1,200 queries across three regions, expanded from real search events using place attributes, reviews, and existing user profiles from the platform. Each query distinguishes mandatory requirements from preferences and factual conditions from subjective judgments, with supporting evidence and criteria for checking each requirement. Queries are validated through iterative assertion checks and cross-model review. We also introduce ATLASBox, a standardized evaluation environment with search tools and a fixed point-of-interest (POI) knowledge base. Given a query, timestamp, and location, an agent searches place attributes and reviews to recommend evidence-supported options or report that no place meets all mandatory requirements. We evaluate answer quality, trajectory quality, and execution cost. Across 14 configurations of 11 models, GPT-6-Luna achieves the highest constraint satisfaction rate among returned recommendations (71.5%) and factual accuracy (95.6%). However, it returns at least one acceptable recommendation for fewer satisfiable queries than GLM-5.3 (67.3% versus 85.3%). These results highlight the challenge of satisfying individual requirements while providing usable recommendations across queries.
Figures & tables
Figure 1. Overview of ATLAS-Bench. (a) Query-only search lacks context and treats all constraints alike. (b) ATLAS-Bench adds context, constraint types, and evidence. (c) Local search within a longer plan. An overview comparing traditional search and agentic local search with multi-faceted constraints.
Dimension
MapEval
LocalSearch
Mobility
MapSatisfy
ATLAS
Query source
Manual
Manual
Trajectory
Map logs
Log-grounded
Reference target
Manual
Manual
Destination
Decision rubrics
Clicked POI
Context
✗
△
✓
✓
✓
Constraint taxonomy
✗
△
✗
△
✓
Constraint conflicts
✗
✗
✗
✗
✓
Implicit needs
✗
△
✗
✓
✓
Table 1. Comparison of local search and mobility benchmarks. ✓ : supported; △ : partial or limited; ✗ : not supported.
Figure 2. Four constraint combinations for a shared restaurant-search intent. A two-by-two matrix illustrating subjective and objective constraints, each expressed as soft preferences or hard requirements.
Figure 3. Overview of ATLAS. (a) Search logs are linked to POI attributes and reviews, privacy-rewritten and validated, and stored as vectors and raw records. (b) Persona-aware queries pass through assertions, cross-model debate, repair, and finalization; failures update the assertion memory from A(t) to A(t+1) . (c) Agents solve the tasks in ATLASBox and are evaluated on answer quality, trajectory quality, and efficiency.
Metric
Chaoyang
Huangpu
Dabancheng
(a) Corpus statistics
Queries
400
400
400
Active POIs
343,916
79,058
2,270
Mean constraints/query
2.17
3.09
2.37
Intent entropy
0.8329
0.6264
0.6750
Constraint combination entropy
0.6608
0.8130
0.5788
Table 2. Regional statistics and constraint coverage. A query may contain multiple constraint types.
Figure 4. Observed synthesis-quality trajectories across ten RSI rounds.
Answer Quality
Trajectory Quality
Efficiency
Model
CSR
CR
Success@k
Recs/q
Factual
Intent
Approp.
Evidence
Strategy
Tokens/q
Calls/q
Rounds/q
Qwen3.5-4B
w/o Think
37.3
65.8
54.0
2.24
40.25
3.53
3.70
1.99
2.96
36.8K
3.79
1.92
w/ Think
44.0
71.4
57.7
1.81
54.33
3.17
3.84
2.11
3.19
50.3K
4.74
2.73
Qwen3.6-35B-A3B
w/o Think
46.5
70.9
73.1
2.43
49.83
3.93
3.99
2.27
3.50
57.6K
6.07
3.33
Table 3. Overall performance on ATLAS-Bench. The best and second-best values are highlighted, excluding efficiency.
Figure 5. Correlations between trajectory and answer quality.
Figure 6. POI distributions, colored by area-normalized density on a shared log scale.
Answer Quality
Trajectory
Efficiency
Region
CSR
CR
Success@k
Recs (/query)
Factual accuracy
Evidence sufficiency
Tool calls
Tokens
Chaoyang
62.9
77.1
80.0
2.81
72.6
2.46
6.04
70.5K
Huangpu
41.4
68.8
63.7
2.08
68.6
1.90
6.12
75.4K
Dabancheng
68.7
84.0
80.8
2.22
81.2
2.91
3.88
43.0K
Table 4. Regional performance across 14 configurations.
C=100
C=200
N
Metric
T=10
T=20
T=30
T=10
T=20
T=30
2
3
4
5
6
7
8
CSR (%)
56.4
60.4
56.4
59.0
59.3
53.3
55.9
52.8
55.8
55.0
58.3
59.1
56.8
CR (%)
75.5
74.4
74.0
76.7
77.5
73.6
77.2
73.5
75.5
74.0
74.1
76.1
73.6
Success@k (%)
74.4
77.9
82.6
76.7
79.1
76.7
73.3
75.6
79.1
80.2
80.2
81.4
77.9
Recs/q
2.24
2.47
2.73
2.34
2.48
2.71
2.45
2.44
2.74
2.52
2.56
2.72
2.60
Factual (%)
81.00
76.00
72.00
80.00
72.00
77.00
79.00
73.00
79.00
77.00
76.00
78.00
76.00
Table 5. Budget sensitivity of Qwen3.8-27B (w/o thinking). C : candidate-pool size; T : results per call; N : maximum rounds.
Method
k
NDCG@10
Hit@10
Recall@20
MRR@20
P50 (s)
BM25
–
0.606
0.813
0.632
0.689
0.059
Vector
100
0.664
0.881
0.675
0.771
0.057
BM25 + rerank
100
0.726
0.884
0.713
0.820
0.564
Vector + rerank
20
0.716
0.899
0.675
0.817
0.160
Vector + rerank
100
0.745
0.927
0.745
0.833
0.568
Vector + rerank
200
0.747
0.935
0.757
0.831
1.078
Table 6. Retrieval comparison across candidate-pool sizes k . Shading marks the default; bold indicates the best value.
Figure 7. Recommendation richness: (a) list-size distributions and (b) quantity versus conditional intra-list diversity.
Figure 8. Contrasting tool-use trajectories: (a) evidence-guided candidate refinement in a high-scoring case; (b) repetitive, off-target retrieval in a failure case.
Figure 9. Long-horizon case with stage choices and evidence.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10. Spatial distributions of the eight most frequent POI categories in Chaoyang, Huangpu, and Dabancheng. Each panel reports a category’s POI count and share of the regional collection. Maps of the leading eight POI categories in each of three benchmark regions, with counts, percentages, and a shared logarithmic density legend.
Figure 11. Condensed prompts for Generate, Debate, and Fix, showing instructions, runtime inputs, and expected outputs.
Figure 12. ATLAS-Bench composition. (a) Shares of 3,049 atomic constraint annotations by dimension. (b) Shares of 1,200 queries by task type, with S1–S8 aggregated. Two donut charts showing 20 constraint dimensions and six task groups. The largest constraint dimensions are location, category, and UGC reputation. Single-constraint queries total 300; M1 through M5 contain 196, 196, 197, 196, and 115 queries.
Benchmark View
Semantic Dist ↑
Surface Div ↑
ATLAS-Bench Pooled
0.5844
0.4622
ATLAS-Bench Beijing
0.5898
0.4067
ATLAS-Bench Shanghai
0.5630
0.4327
ATLAS-Bench Urumqi
0.5338
0.2854
LocalSearchBench Question
0.4946
0.0629
LocalSearchBench Hop
0.5830
0.1697
Appendix
Table 7. Comparison of semantic and surface diversity at 400 instances.
Check
Observed issue
Revision
Name leakage
Geographic anchors falsely flagged.
Exempt geographic terms; fix whitespace matching.
Evidence linkage
POI-based joins mixed search events.
Join evidence by source-record ID.
Semantic rejection
Lexical rules rejected valid queries.
Broaden matching; make the check advisory.
Conflict references
Mixed indexing linked wrong constraints.
Normalize index conventions.
Constraint types
Overlapping fields conflated dimensions.
Refine matching; separate semantic dimensions.
Satisfiability
Default labels masked conflicts.
Remove positive defaults; repair conflict logic.
Appendix
Table 8. Representative assertion and validation revisions.
Tool
Purpose
Inputs
search_pois
Retrieve POIs and structured attributes; resolve landmark coordinates.
query
search_notes
Retrieve review evidence for experiential requirements.
query
get_poi_notes
Inspect reviews linked to a specific POI.
poiid
search_nearby
Find POIs within a radius of the given coordinates.
query, loc, distance_m
Appendix
Table 9. ATLASBox search tools and required inputs. Optional arguments control result limits and filters.
LLM recommendation agents increasingly produce structured recommendation reports: sets of items accompanied by natural-language justifications. Yet existing evaluations often reduce this setting to reranking small shortlisted candidate sets or judge reports mainly by semantic plausibility. We introduce Recommendation Atlas (Agentic Tool-Level Assessment for Shopping), or RecoAtlas, a benchmark and toolkit for evaluating shopping agents with behavior-grounded metrics. RecoAtlas complements held-out interaction metrics with learned utility proxies for relevance, complementarity, and diversity derived from interaction data, while separately measuring semantic coherence and explanation quality. Its controlled tool environment exposes agents to either semantic, behavior-aligned, or faulty tools, enabling diagnosis of whether performance gains arise from stronger reasoning, better signals, or more effective tool-use policies. Across controlled experiments, we show that RecoAtlas exhibits key properties of a meaningful benchmark for agentic systems: performance scales with model capacity and test-time compute, improves with stronger and better-aligned tools, degrades under noisy or misaligned signals, and reveals that semantic plausibility does not necessarily capture behavior-grounded utility. RecoAtlas provides a foundation for developing and evaluating shopping assistants that optimize not only for plausible recommendations, but also for coherent, behaviorally grounded recommendation sets.
Agentic search has emerged as a promising paradigm for adaptive retrieval systems powered by large language models (LLMs). However, existing benchmarks primarily focus on quality, overlooking efficiency factors that are critical for real-world deployment. Moreover, real-world user queries often contain underspecified preferences, a challenge that remains largely underexplored in current agentic search evaluation. As a result, many agentic search systems remain impractical despite their impressive performance. In this work, we introduce HotelQuEST, a benchmark comprising 214 hotel search queries that range from simple factual requests to complex queries, enabling evaluation across the full spectrum of query difficulty. We further address the challenge of evaluating underspecified user preferences by collecting clarifications that make annotators' implicit preferences explicit for evaluation. We find that LLM-based agents achieve higher accuracy than traditional retrievers, but at substantially higher costs due to redundant tool calls and suboptimal routing that fails to match query complexity to model capability. Our analysis exposes inefficiencies in current agentic search systems and demonstrates substantial potential for cost-aware optimization.
LLM search agents are often evaluated on final-answer accuracy, overlooking the process. Analyzing a search strategy requires understanding how credible evidence is retrieved to address question constraints. This valuable information is buried in raw search trajectories that are long and difficult to parse. We introduce SearchAtlas, a framework that converts search trajectories into structured graphs whose edges represent how evidence is propagated across the reasoning trace, from the query that retrieves it to the final answer. Our automated parsing pipeline achieves a mean edge F1 of 86.0% against human-annotated graphs and remains consistent across repeated runs. We analyze five search agents on three benchmarks, revealing systematic differences in search scale and evidence aggregation. SearchAtlas exposes fragmented answer support, question constraints that do not reach the answer, and unverified parametric knowledge entering the response. These process failures are strongly associated with incorrect answers, even more so than an LLM judge given either the raw trajectory or the ordered query list, suggesting that the constructed graphs provide useful interpretability. Moreover, an audit of cases in which process-diagnostic scores disagree with final-answer correctness shows that they capture information not reducible to answer accuracy.