PriceBench: A Diagnostic Benchmark for Price, Quality, and Brand Preferences in LLM Booking Agents

Pavel Kireyev

econ.GNSep 25, 2026

PriceBench: A Diagnostic Benchmark for Price, Quality, and Brand Preferences in LLM Booking Agents

Authors: Pavel Kireyev

Organizations: Department of Management London School of Economics and Political Science

Abstract

LLMs increasingly act as purchasing agents, which makes the LLM, not the user, the one choosing among the options that satisfy a request; its preferences quietly fix what gets bought and what it costs. Hotel booking is a clean instance: a high-volume choice settled on a few comparable attributes, where the pick reveals those preferences. We introduce PriceBench, a diagnostic benchmark that recovers an LLM's price, quality, and brand preferences from its booking choices with a logit choice model, applied to 28 LLMs from 8 providers on 3,600 hotel tasks from 179 real New York City properties. We find that capability is associated with how consistently an LLM chooses, not with what it chooses: more capable LLMs hold stronger, more consistent preferences, while weaker ones either lock onto one position, exploitable by whoever controls listing order, or choose almost indifferently. What those preferences favor varies sharply across providers and even within one family: price sensitivity spans more than an order of magnitude, and the price/quality trade-off moves mean booked nightly price from $247 to $393 on identical tasks. What an agent buys must therefore be measured per LLM, not inferred, and we release the tasks, code, and all 28 response sets.

Figures & tables

Panel of Figure 1 — Figure 1: Task 113 from PriceBench: which option do you choose? GPT-5.4, GPT-5.4 Mini, GPT-4.1 Mini, Claude Haiku 4.5, and Gemma3 27B take A; Gemma3 4B, DeepSeek-R1 7B, and Phi-4 Mini take B. Different agents, different revealed preferences.

Figure 2: Per-LLM price response. The $y$ -axis is each price decile’s choice utility relative to the cheapest decile, in logit (log-odds) units: $0$ = as likely to be chosen as the cheapest rooms, more negative = less likely; all three estimators are pinned to $0$ there. Black points are non-parametric decile estimates ( $\pm 1$ SE band); the log-price (purple) and linear-price (orange dashed) curves are fit to the raw choices, not to the 10 decile points, so they need not pass through them, and the AIC winner is drawn bold. Each panel reports, bottom-left, $\beta_{\ln p}$ , $\beta_{p}$ , and $\Delta\mathrm{AIC}=\mathrm{AIC}_{\text{lin}}-\mathrm{AIC}_{\text{log}}$ (negative = linear wins); panels are sorted by $|\beta_{\ln p}|$ , most sensitive first. Twenty of the 21 engaged LLMs with a genuine negative price response are shown; the weakest, Qwen3 30B-A3B ( $\beta_{\ln p}=-0.40$ , AIC favors log), is omitted for space but appears in the full 23-panel grid (Appendix C.1 ), with the two flat-response LLMs.

Figure 3: Price–quality map. $x$ -axis: $|\beta_{\ln p}|$ (price sensitivity, holding quality fixed). $y$ -axis: $\beta_{\text{review\_score}}$ (quality sensitivity, holding price fixed). Colored by provider; dashed lines at the median split the plane into four regions. Distance from the origin is largely decisiveness (the logit scale inflates every coefficient at once), which is why most LLMs fall on the diagonal; the scale-free price/quality trade-off is shown in Figure 9 (Appendix D ).

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: First-shown selection rate per LLM, pooled over both orderings of all 3,600 tasks (exact construction in the text). Bars in $[15\%,85\%]$ are engaged and colored by provider; bars outside the band are position-locked (grey). The $50\%$ reference line marks balanced choice.

Figure 5: Non-parametric price-response curves for all 23 engaged LLMs, shared $y$ -axis, panels sorted by $|\beta_{\ln p}|$ from the parametric log-price fit. Each point is the utility of a given price decile relative to the cheapest decile (decile 1 is the reference, fixed at $0$ ). $R^{2}$ values (McFadden pseudo- $R^{2}$ ) are computed from the log-price fit on the same data.

Figure 6: Binary (blue) vs. ternary (orange) log-price (left) and linear-price (right) coefficients per engaged LLM, $\pm 1$ SE. Spearman rank correlation: $\rho=0.88$ for log-price; $\rho=0.90$ for linear-price.

Figure 7: Binary log-price sensitivity $|\beta_{\ln p}|$ against (log) parameter count for engaged LLMs with a significant price effect. Filled circles: open-weight (real parameter counts). Open squares: proprietary LLMs, placed on tier-based ordinal estimates derived from API pricing ladders. Thin lines connect same-family LLMs.

Figure 8: Quality coefficients per engaged LLM from the per-LLM binary choice model. Rows sorted by price sensitivity $|\beta_{\ln p}|$ (sharpest at top). Cell number = raw logit coefficient; cell color = within-column $z$ -score across LLMs, isolating relative position on each quality dimension from overall coefficient scale.

Figure 9: Scale-free price/quality trade-off: willingness to pay for one review-score point, $(\beta_{\text{review\_score}}/|\beta_{\ln p}|)\times\$250$ , where $250 is a typical reference nightly rate, for the 20 engaged LLMs with both coefficients significant at $p<0.001$ . Bars colored by provider; error bars are approximate $\pm 1$ SE (delta method). Low end = bargain hunters, high end = quality chasers. The ranking is uncorrelated with decisiveness (pseudo- $R^{2}$ ; Pearson $r=-0.09$ ). — Figure 9: Scale-free price/quality trade-off: willingness to pay for one review-score point, $(\beta_{\text{review\_score}}/|\beta_{\ln p}|)\times\$ 250 $, where$ 250 is a typical reference nightly rate, for the 20 engaged LLMs with both coefficients significant at $p<0.001$ . Bars colored by provider; error bars are approximate $\pm 1$ SE (delta method). Low end = bargain hunters, high end = quality chasers. The ranking is uncorrelated with decisiveness (pseudo- $R^{2}$ ; Pearson $r=-0.09$ ).

Specification	Wyndham	All-5 pos.	Perm. $p$
Baseline	$+2.17$	11/23	0.48
Drop top-leverage listing	$+2.07$	12/23	0.47
Area controls	$+1.23$	7/23	0.42
Luxury-tier indicator	$+2.18$	11/23	0.63

Appendix

Table 1: Brand robustness, one specification per row: GPT-5.4’s Wyndham coefficient (the largest chain premium), the number of engaged LLMs preferring all five chains to an independent, and the one-sided provider-null permutation

p

. All Wyndham coefficients shown are significant at

p<0.001

Figure 10: Brand coefficients (relative to Independent) from a logit controlling for price deciles, stars, room type, cancellation, breakfast, review score, and review count. Positive = the LLM prefers this chain over an Independent with identical controls. Rows sorted by total magnitude of brand response. Several capable LLMs over-select Wyndham (GPT-5.4

+2.17

, GPT-5.4 Mini

+1.70

, Gemma3 27B

+1.15

, GPT-4.1 Mini

+1.06

); the premium is spread across the pool’s five Wyndham properties rather than driven by any one listing (leave-one-property-out moves the GPT-5.4 coefficient by at most

0.21

; Table 1 ). Claude Haiku 4.5 is distinctive in spreading its preference evenly across all five chains (minimum

+0.60

, mean

+0.71

LLM	Variant	$\beta_{\ln p}$ (SE)	$\beta_{\text{review}}$ (SE)	WTP (SE)	pseudo- $R^{2}$
Qwen3 4B	baseline	$-1.56$ (0.14)	$+0.80$ (0.11)	128 (20)	0.129
	JSON	$-0.39$ (0.11)	$+0.32$ (0.09)	203 (84)	0.013
	reorder	$-1.49$ (0.14)	$+0.80$ (0.11)	134 (22)	0.141
	persona	$-1.79$ (0.14)	$+1.07$ (0.11)	150 (20)	0.176
	CoT	$-1.99$ (0.16)	$+0.45$ (0.12)	56 (15)	0.296
Gemma2 9B	baseline	$-5.56$ (0.30)	$+1.56$ (0.16)	70 (8)	0.516

Appendix

Table 2: Prompt-format ablation: log-price logit per LLM and variant, both orderings, temperature 0, baselines re-estimated on the variants’ task subsets (

n=1{,}800

responses per cell; 900 responses for Gemma3 27B’s baseline). WTP is willingness to pay for one review-score point at the $250 reference, in dollars. Phi-4 Mini’s JSON WTP is omitted because its price coefficient is indistinguishable from zero, so the ratio is unidentified.

LLM	Role	First ( $T{=}0$ )	First ( $T{=}0.7$ )	$\beta_{\ln p}$ at 0.7 (SE)	$\beta_{\ln p}$ at 0	pseudo- $R^{2}$ (0.7 / 0)
Llama3.2 1B	locked	100.0%	99.3%	$+0.00$ (0.08)	$-0.00$	0.001 / 0.000
Mistral 7B	locked	99.5%	98.9%	$-0.08$ (0.08)	$-0.00$	0.001 / 0.000
Qwen3 0.6B	locked	98.5%	52.2%	$-0.17$ (0.08)	$-0.01$	0.031 / 0.002
Llama3 8B	locked	88.2%	91.4%	$-0.69$ (0.08)	$-0.59$	0.037 / 0.029
Llama3.1 8B	locked	88.2%	89.2%	$-0.73$ (0.08)	$-0.55$	0.040 / 0.022
Gemma3 4B	engaged control	66.2%	58.9%	$-2.51$ (0.12)	$-1.33$	0.280 / 0.216

Appendix

Table 3: First-shown rates and re-fit price coefficients at temperature 0.7 (

n=3{,}600

binary responses per LLM). The

T{=}0

column repeats the pooled triage rate of Figure 4 ; the

T{=}0.7

rates are on the binary block alone, where the two Llama 8B variants already sit at 92.6% and 93.3% at

T{=}0

, so sampling does not raise their rates.

LLM	First-slot	Hit	Top-2	Mean pred. rank	$\beta_{\ln p}$ five-opt (SE)
Gemma2 9B	10.2%	52.2%	84.2%	1.67	$-3.49$ (0.25)
Qwen3 4B	1.8%	50.7%	78.0%	1.80	$-2.61$ (0.21)
Phi-4 Mini	6.3%	44.3%	69.8%	2.04	$-0.48$ (0.15)
Mistral-Nemo 12B	55.5%	46.7%	67.7%	2.12	$-1.82$ (0.18)
Llama3.1 8B	61.2%	42.3%	66.2%	2.12	$-1.81$ (0.17)

Appendix

Table 4: Five-option pilot (

n=600

responses per LLM). First-slot = share of picks on the first-listed option; Hit = the binary-estimated utilities’ top prediction is chosen (chance 20%); Top-2 = the choice falls in the predicted top two (chance 40%); mean predicted rank of the chosen option has chance value 3.0. The last column re-fits the log-price coefficient on the five-option choices themselves.

LLM	$\beta_{\ln p}$ [95% CI]	WTP [95% CI]	Rank $\|\beta_{\ln p}\|$ [CI]	Rank WTP [CI]
GPT-5.4	$-7.02$ [ $-8.08$ , $-6.29$ ]	107 [94, 121]	1 [1, 1]	19 [18, 19]
Gemma2 9B	$-5.71$ [ $-6.24$ , $-5.27$ ]	68 [59, 78]	2 [2, 3]	20 [20, 20]
GPT-4.1 Mini	$-5.68$ [ $-6.35$ , $-5.15$ ]	119 [108, 131]	3 [2, 3]	18 [17, 19]
Claude Haiku 4.5	$-4.75$ [ $-5.28$ , $-4.31$ ]	139 [122, 157]	4 [4, 6]	16 [16, 17]
GPT-5.4 Mini	$-4.56$ [ $-5.18$ , $-4.11$ ]	222 [203, 246]	5 [4, 6]	12 [11, 14]
Gemma3 27B	$-4.38$ [ $-4.91$ , $-3.92$ ]	210 [191, 233]	6 [4, 6]	13 [12, 15]

Appendix

Table 5: Cluster-bootstrap 95% percentile intervals (

B=500

, resampling the 450 binary choice pairs whole) for the log-price coefficient and the willingness to pay for one review-score point at the

250 reference (dollars), with rank intervals. Price-sensitivity ranks cover the 23 engaged LLMs (1 = most price-sensitive); WTP ranks cover the 20-LLM well-identified screen (1 = highest WTP). The three LLMs outside the screen (not both coefficients significant at

p<0.001$ ) have no meaningful WTP and are dashed out.

Study	Setting and construct	Conversion	Per point at $250
Anderson (2012) (research-center report)	Travelocity purchase logit; price change holding purchase probability constant	11.2% per 5-pt point $\div$ 2 = 5.6%	$14.00
Anderson (2012) , same report	GRI-to-ADR equilibrium regression	0.89% per 1% of GRI $\times$ 10 = 8.9% (upper bound)	up to $22.25
Lewis and Zervas (2016) (working paper)	Five-state hotel panel; price regression, hotel fixed effects	1.5% (pooled) to 9% (2014) per 5-pt point $\div$ 2	$1.88–$ 11.25
Ursu (2018)	Expedia, randomized ranking; sequential search model	review coefficient negative or insignificant, all four destinations	$\leq$ $0
Ghose et al. (2012)	Travelocity transactions; random-coefficients logit, log price	ratio of review-rating to log-price coefficients (our derivation; the conversion equation of Ghose et al., 2014 independently corroborates the ratio, 0.237 against 0.228)	$29–$ 33 per platform; $61 summed
Chen and Yao (2017)	US OTA clickstream; dollar-denominated utility	entire above-4.5 versus below-4 band worth $99.11 per night	$\leq$ $99 (band, not point)

Appendix

Table 6: Human anchors for the value of one review-score point (10-point scale) at a $250 nightly rate. The first three rows are equilibrium price movements; the next three are structural estimates from real bookings; the last two are stated-choice and hedonic context at coarser units. Demand-side evidence ( Öğüt and Taş, 2012 ; Viglia et al., 2016 ) supports the same direction but is not convertible to dollars per point.

LLM Agents

Large language model agents use language models to plan, use tools, and act across multi-step tasks and environments.

See all 685 papers

Whose hotel does the AI recommend? An algorithm audit of reputation signals in LLM-assisted hotel selection
Jun 15, 2026Mirza Samad Ahmed Baig, Syeda Anshrah Gillani, Asher AliLLM Auditing Recommender Systems
Fandaqah, Al Khobar, Saudi Arabia. · Hamdard University, Karachi, Pakistan.
Can LLM Agents Price Competitively? A Dynamic Multi-Attribute Auction Benchmark for Agentic Commerce
Jul 30, 2026Shimaa Ahmed, Yiwei Cai, Mohsen Minaei +1LLM Agent Evaluation AI Agent Benchmarks
Visa Research
PrefBench: Evaluating Zero-Shot LLM Agents in Hidden-Preference Personalized Pricing Negotiations
May 19, 2026Yingjie LeiAutomated Negotiation LLM Agent Evaluation
University of Aberdeen

Jun 15, 2026J/K move · Enter open · S save

Whose hotel does the AI recommend? An algorithm audit of reputation signals in LLM-assisted hotel selection

Mirza Samad Ahmed Baig, Syeda Anshrah Gillani, Asher Ali

Fandaqah, Al Khobar, Saudi Arabia. · Hamdard University, Karachi, Pakistan.

LLM Auditing Recommender Systems Large Language Model-Based Recommendation Position Bias

Open paper AskPDF arXiv

Travelers increasingly ask large language model (LLM) assistants which hotel to book, making these systems gatekeepers of property visibility -- yet what moves their recommendations is undocumented. We conduct a pre-specified algorithm audit using a randomized choice-based conjoint: across personas, prompt templates, and twelve open-weight and proprietary models, assistants choose among five hotels whose guest rating, review volume and recency, management response, chain affiliation, price, eco-certification, and list position are independently randomized. We estimate the average marginal component effect of each signal on the probability of recommendation. Guest rating and price dominate (a top rating raises selection by 31.6 percentage points; a high price lowers it by 30.0), reproducing human valence-and-price primacy but over-weighting eco-certification and ignoring management response. List position -- a content-free artifact -- shifts recommendations causally, worth about $12 per night. Stated reasons track revealed weights imperfectly. The findings ground generative engine optimization and the accountability of AI infomediaries in causal evidence.

Abstract

LLM Agents

Explore similar work

Whose hotel does the AI recommend? An algorithm audit of reputation signals in LLM-assisted hotel selection

Can LLM Agents Price Competitively? A Dynamic Multi-Attribute Auction Benchmark for Agentic Commerce

PrefBench: Evaluating Zero-Shot LLM Agents in Hidden-Preference Personalized Pricing Negotiations