HotelQuEST: Balancing Quality and Efficiency in Agentic Search

Guy Hadad; Shadi Iskander; Oren Kalinsky; Sofia Tolmach; Ran Levy; Haggai Roitman

cs.IRFeb 27, 2026

HotelQuEST: Balancing Quality and Efficiency in Agentic Search

Authors: Guy Hadad, Shadi Iskander, Oren Kalinsky, Sofia Tolmach, Ran Levy, Haggai Roitman

Organizations: Ben-Gurion University · Amazon

Abstract

Agentic search has emerged as a promising paradigm for adaptive retrieval systems powered by large language models (LLMs). However, existing benchmarks primarily focus on quality, overlooking efficiency factors that are critical for real-world deployment. Moreover, real-world user queries often contain underspecified preferences, a challenge that remains largely underexplored in current agentic search evaluation. As a result, many agentic search systems remain impractical despite their impressive performance. In this work, we introduce HotelQuEST, a benchmark comprising 214 hotel search queries that range from simple factual requests to complex queries, enabling evaluation across the full spectrum of query difficulty. We further address the challenge of evaluating underspecified user preferences by collecting clarifications that make annotators' implicit preferences explicit for evaluation. We find that LLM-based agents achieve higher accuracy than traditional retrievers, but at substantially higher costs due to redundant tool calls and suboptimal routing that fails to match query complexity to model capability. Our analysis exposes inefficiencies in current agentic search systems and demonstrates substantial potential for cost-aware optimization.

Figures & tables

Figure 1: Illustration of a task from our benchmark.

Name	Domain	Size	Language	Complexity	A	F	E
Mind2Web 2 Gou et al. (2025)	General	130	English	High	$\checkmark$	$\checkmark$	$\times$
WebShop Yao et al. (2022)	E-Commerce	12,087	English	Low	$\checkmark$	$\times$	$\times$
BrowseComp Wei et al. (2025)	General	1,266	English	High	$\checkmark$	$\times$	$\times$
TheAgentCompany Xu et al. (2024)	Enterprise	175	English	Undefined	$\checkmark$	$\times$	$\checkmark$
GAIA Mialon et al. (2023)	General assistant	466	English	Low to High	$\checkmark$	$\times$	$\times$
GAIA2 Andrews et al. (2025)	General	963	English	High	$\checkmark$	$\checkmark$	$\times$

Table 1: Comparison between benchmarks. Top: agentic benchmarks involving search among other requirements across general, e-commerce, and enterprise domains. Middle: agentic search benchmarks focusing on deep research, factual seeking, and broad search. Columns A , F , and E indicate Accuracy , Factuality , and Efficiency , respectively.

Section	Model	Subset	Quality		Efficiency
			Accuracy	Factuality	Cost ($)	#Tokens	P50 (s)	P90 (s)
Retrieval only	BM25	Reviews	2.64	–	0.00	–	0.23	0.23
		Descriptions	1.80	–	0.00	–	0.0046	0.0046
	Dense (22M)	Reviews	2.56	–	0.00	–	0.0007	0.0007
		Descriptions	2.22	–	0.00	–	0.0087	0.0087
	Dense (300M)	Reviews	3.00	–	0.00	–	0.0054	0.0054

Table 2: Evaluation split into Retrieval only , Retrieval + LLM-based Reranker , and LLM-based Agents on Reviews and Descriptions , as well as two versions of Oracle models. Metrics cover Quality and Efficiency .

Figure 2: Accuracy–Efficiency Trade-off. Numbers above each point indicate the agent’s iteration limit. Left: As cost increases, accuracy initially improves, but beyond a certain point additional cost yields no further gains. Middle: A similar pattern appears with median latency: accuracy rises with longer deliberation until both metrics plateau and converge. Right: The P90 latency curve mirrors the cost trend, indicating that on some queries the model fails to terminate early, leading to disproportionately high latency and cost.

Figure 3: Budget Oracle. Accuracy achieved by solving a multiple-choice knapsack problem under varying budget limits. A clear elbow appears around $2.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Distribution of query lengths.

Qualifier Type	Qualifier Content
Explicit / Implicit	Purpose
Negation	Location
Similarity	Population
Range	Seasonality
Time-sensitive	Description
Optional / Mandatory	Rating

Appendix

Table 3: Taxonomy of qualifier types and contents.

Figure 5: Analysis of the queries by the presence of qualifier attributes.

Figure 6: Illustration of the agentic workflow.

	Simple				Moderate				Complex
Metric	Qwen3-32B	Sonnet 4	Haiku 4.5	Sonnet 3.7	Qwen3-32B	Sonnet 4	Haiku 4.5	Sonnet 3.7	Qwen3-32B	Sonnet 4	Haiku 4.5	Sonnet 3.7
Cost	0.022	0.212	0.078	0.482	0.025	0.226	0.095	0.479	0.026	0.250	0.092	0.461
Tokens	77K/18K	33K/8K	31K/8K	73K/18K	83K/20K	36K/8K	37K/10K	73K/17K	90K/21K	39K/9K	35K/10K	70K/17K
Latency (sec)	83.78	136.60	74.36	425.95	89.99	141.24	91.42	423.81	96.98	156.88	93.02	416.82
Accuracy	4.135	4.275	3.519	4.423	3.613	3.974	3.613	4.150	3.850	4.093	3.550	4.175

Appendix

Table 4: Metrics by complexity level and model; tokens shown as inputK/outputK.

Figure 8: Quality Oracle. Distribution of selected models when choosing, for each query, the cheapest model among those achieving the highest accuracy.

Score	Label	Description	Criteria
5	Exact Match	The answer completely addresses all aspects of the query with specific, actionable hotel recommendations.	$\bullet$ Addresses all requirements (location, budget, amenities, group size, etc.)
			$\bullet$ Provides specific hotel names and relevant details
			$\bullet$ Explains why each recommendation fits
4	Strong Match	Covers almost all requirements, with minor omissions or slight generalization.	$\bullet$ Addresses most requirements with relevant hotels
			$\bullet$ Missing minor detail (e.g., exact price or a less critical amenity)
3	Partial Match	Covers some requirements but misses key aspects.	$\bullet$ Addresses some requirements

Appendix

Table 5: Accuracy scoring rubric for hotel recommendation answers.

Figure 9: Confusion matrices measuring alignment between human judgments and LLM-based scoring across different levels of label aggregation.

Setting	Exact Match	$\kappa_{\text{linear}}$	$\kappa_{\text{quad}}$	$\rho$	$\tau$	$r$	Wilcoxon $p$	t-test $p$
Original (1–5)	0.5813	0.701	0.841	0.844**	0.755**	0.851**	**	**
Agg.: 1 \| 2–3 \| 4–5	0.7683	0.721	0.796	0.808**	0.767**	0.811**	**	**
Agg.: 1–2 \| 3 \| 4–5	0.7805	0.742	0.810	0.817**	0.769**	0.817**	**	**
Agg.: 1–2 \| 3–4 \| 5	0.7317	0.681	0.774	0.785**	0.734**	0.777**

Appendix

Table 6: Agreement between human annotators and LLM-as-a-judge ratings. ** denotes

p<0.01

; * denotes

p<0.05

Explore similar work

When Search Agents Should Ask: DiscoBench for Clarification-Aware Deep Search
Jun 26, 2026Yiling Tao, Shihan Deng, Meiling Tao +3Agentic Search LLM Agent Evaluation
Hunyuan, Tencent · Shenzhen International Graduate School, Tsinghua University
Quantifying the Accuracy and Cost Impact of Design Decisions in Budget-Constrained Agentic LLM Search
Mar 9, 2026Kyle McCleary, James GhawalyAgentic Search Agentic RAG
Division of Computer Science & Engineering, Louisiana State University, Baton Rouge, Louisiana, USA
ScholarQuest: A Taxonomy-Guided Benchmark for Agentic Academic Paper Search in Open Literature Environments
Jun 18, 2026Tingyue Pan, Mingyue Cheng, Daoyu Wang +4Agentic Search Information Retrieval
State Key Lab of Cognitive Intelligence, University of Science and Technology of China

Jun 26, 2026J/K move · Enter open · S save

When Search Agents Should Ask: DiscoBench for Clarification-Aware Deep Search

Yiling Tao, Shihan Deng, Meiling Tao, Pengzhi Wei+2

Hunyuan, Tencent · Shenzhen International Graduate School, Tsinghua University

Agentic Search LLM Agent Evaluation Conversational Search AI Agent Benchmarks

Open paper AskPDF arXiv

Search agents powered by large language models (LLMs) are increasingly used to solve complex information-seeking tasks, requiring multi-step retrieval and reasoning to fulfill user goals. However, existing benchmarks often assume that user queries are complete and explicit, overlooking the fact that real-world search requests are frequently vague, underspecified, or even factually incorrect. In deep search scenarios, such ambiguity can propagate along multi-step reasoning chains and lead agents toward incorrect search trajectories. To address this gap, we introduce DiscoBench, a benchmark for clarification-aware deep search, designed to evaluate whether search agents can proactively identify ambiguity, ask effective clarification questions, and recover correct reasoning paths through user interaction. DiscoBench contains 211 samples and 463 ambiguity instances across 11 real-world domains, covering four ambiguity types. We further design a user simulator for multi-turn interaction and evaluate model performance from four perspectives: task utility, ambiguity detection, interaction strategy, and cost efficiency. Experiments on representative LLMs show that ambiguity detection and effective clarification are distinct capabilities, and that repeatedly searching instead of asking for clarification often performs worse than direct guessing, highlighting a critical gap between retrieval ability and interactive problem-solving in current search agents.

Abstract

Explore similar work

When Search Agents Should Ask: DiscoBench for Clarification-Aware Deep Search

Quantifying the Accuracy and Cost Impact of Design Decisions in Budget-Constrained Agentic LLM Search

ScholarQuest: A Taxonomy-Guided Benchmark for Agentic Academic Paper Search in Open Literature Environments