Vendster: An Evaluation Arena to Investigate How Forecasting Tool Quality Affects LLM Agent Behaviors in Buyer-Vendor Negotiation

Charin Polpanumas

cs.GTOct 28, 2026

Vendster: An Evaluation Arena to Investigate How Forecasting Tool Quality Affects LLM Agent Behaviors in Buyer-Vendor Negotiation

Authors: Charin Polpanumas

Organizations: Amazon, Japan

Abstract

Time-series forecasters are ranked almost entirely by accuracy metrics, but a forecast's decision value, its effect on the downstream decision, need not track accuracy and is rarely measured. This problem compounds as large language model (LLM) agents increasingly consume forecasters as tools, mediating them through stochastic, opaque policies. We present Vendster, an evaluation arena measuring how a forecaster's bias and prediction-interval width change the outcome of a buyer--vendor discount negotiation between two LLM agents. In the reference instance Vendster-M5, sampling base demand patterns from the M5 dataset, we evaluate two frontier-closed LLM families (Claude Opus 4.6 and GPT-5.6 Sol) and two open ones (DeepSeek-V3.2 and Qwen3-235B-A22B). Model-generated forecasts raise sales target attainment by up to +17 percentage points (p<0.001), yet standard forecast evaluation metrics such as relWQL and MASE do not predict negotiation outcomes at a statistically significant level. To isolate the causal structure, we substitute in synthetic forecasts with controlled bias and interval width. Under-prediction bias significantly improves outcomes for agents that react more to the forecasts---dose-response slopes -0.387 and -0.416 (p<0.001). Interval width has no statistically significant effect at zero bias. These findings indicate that the direction of forecast bias and agent forecast-reactivity are more strongly associated with downstream outcomes than forecast accuracy. Vendster is fully customizable and released under CC BY-NC 4.0 at https://github.com/amazon-science/Vendster.

Figures & tables

			Causal		Observational
LLM	No Forecast	Forecast Uplift	Bias	Width	Bias	Width	relWQL
Opus	0.24	$+0.12^{*}$ (.000, .000)	$-0.387^{*}$ (.000, .000)	$+0.016$ (.225, .899)	$-0.466^{*}$ (.009, .034)	$+0.000$ (.904, 1.000)	$-0.021$ (.703, 1.000)
GPT	0.25	$+0.17^{*}$ (.000, .000)	$-0.416^{*}$ (.000, .000)	$+0.002$ (.882, 1.000)	$-0.408$ (.057, .171)	$-0.002$ (.053, .211)	$-0.126$ (.016, .062)
Qwen	0.11	$+0.11^{*}$ (.000, .000)	$+0.001$ (.988, 1.000)	$-0.003$ (.796, 1.000)	$-0.195$ (.266, .510)	$+0.001$ (.423, 1.000)	$+0.009$ (.835, 1.000)
DeepSeek	0.22	$+0.08^{*}$ (.001, .001)	$+0.012$ (.855, 1.000)	$-0.002$ (.867, 1.000)	$-0.182$ (.255, .510)	$+0.001$ (.574, 1.000)	$+0.044$ (.362, 1.000)

Table 1: Forecast effects by LLM. Forecast Uplift is the H0 uplift in sales target attainment rate from model forecasts over Buyer agent with no forecast. Causal columns are slopes on the varied bias/width grid; Observational columns regress attainment on measured bias, width, and relWQL. Cells show effect

(\text{raw}\ p,\ \text{Holm}\ p)

; ∗ : Holm

p<0.05

across the four LLMs.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

metric	Opus	GPT	Qwen	DeepSeek
relWQL	$-0.021$ (.703, 1.000)	$-0.126$ (.016, .062)	$+0.009$ (.835, 1.000)	$+0.044$ (.362, 1.000)
MASE	$-0.095$ (.362, 1.000)	$-0.194$ (.104, .415)	$+0.005$ (.961, 1.000)	$+0.055$ (.510, 1.000)
MAPE	$-0.156$ (.535, 1.000)	$-0.353$ (.261, .783)	$+0.015$ (.953, 1.000)	$+0.308$ (.154, .616)
sMAPE	$-0.195$ (.598, 1.000)	$-0.408$ (.424, 1.000)	$+0.098$ (.765, 1.000)	$+0.387$ (.147, .590)
RMSE	$-0.003$ (.453, 1.000)	$-0.007$ (.122, .488)	$+0.001$ (.854, 1.000)	$+0.003$ (.488, 1.000)
CRPS	$-0.004$ (.477, 1.000)	$-0.011$ (.068, .270)	$+0.003$ (.625, 1.000)	$+0.006$ (.363, 1.000)

Appendix

Table 2: Forecast quality

\to

sales target attainment: per-basket slope by metric

\times

model, cell

=

slope

(\text{raw}\ p,\ \text{Holm}\ p)

; ∗ : Holm

p<0.05

		first-offer	agreed disc.	concession	turns	agreement
Opus	presence	$+0.046^{*}$ (.000, .000)	$+0.030^{*}$ (.000, .000)	$+0.005^{*}$ (.000, .000)	$+1.03^{*}$ (.000, .000)	$+0.003$ (.333, .429)
Opus	bias slope	$-0.101^{*}$ (.000, .000)	$-0.082^{*}$ (.000, .000)	$+0.002$ (.341, 1.000)	$-3.09^{*}$ (.000, .000)	$+0.002$ (.544, .928)
GPT	presence	$+0.020^{*}$ (.000, .000)	$+0.053^{*}$ (.000, .000)	$-0.016^{*}$ (.000, .000)	$+6.61^{*}$ (.000, .000)	$-0.135^{*}$ (.000, .000)
GPT	bias slope	$-0.139^{*}$ (.000, .000)	$-0.195^{*}$ (.000, .000)	$+0.047^{*}$ (.000, .000)	$-21.94^{*}$ (.000, .000)	$+0.926^{*}$ (.000, .000)
Qwen	presence	$+0.105^{*}$ (.000, .000)	$+0.041^{*}$ (.000, .000)	$-0.003$ (.168, .168)	$-1.68^{*}$ (.000, .000)	$+0.155^{*}$ (.000, .000)
Qwen	bias slope	$+0.026^{*}$ (.003, .007)	$-0.004$ (.577, 1.000)	$-0.001$ (.639, 1.000)	$+0.08$ (.906, .906)	$+0.038$ (.464, .928)

Appendix

Table 3: Forecast-reactivity behaviors. Columns are the logged negotiation behaviors: first-offer (mean discount in the Buyer’s opening proposal), agreed disc. (mean discount at agreement), concession (mean absolute change between a proposer’s successive proposal means), turns (turns to termination), and agreement (deal rate). For each family, two rows: presence —a behavior’s uplift from having a forecast—and bias slope —its per-basket slope along the varied bias ladder. Cells show the statistic over

(\text{raw}\ p,\ \text{Holm}\ p)

; ∗ : Holm

p<0.05

Figure 1: Sales target attainment against synthetically varied bias (top) and interval width (bottom), one column per model, with 95% confidence intervals.

Figure 2: Bias-attainment slope by using different LLMs for Buyers and Vendors; this suggests the slope is Buyer-dependent.

Figure 3: Operating-point selection using the no-forecast Sonnet 4.6 agent, , with 95% confidence intervals.

Hypothesis	effect tested	power ( $\Delta{=}0.10$ )	floor (attain)
H0	aggregate forecast vs no_forecast	0.81	—
H1c	bias, causal grid	0.98	0.070
H2c	width, causal grid	0.94	0.080
H1o	bias, observational	0.88	0.089
H2o	width, observational	0.97	0.072
H3o	quality (relWQL), observational	0.97	0.074

Appendix

Table 4: A-priori power at the locked design (

K{=}16

m{=}25

\alpha{=}0.05

) for the

0.10

-attainment MDE. Power : probability of detecting a

0.10

effect; floor : minimum detectable effect at

80\%

power. H0 is sized directly to the MDE; slope power is range-independent, so the observational tests size despite the forecasters’ tight spread. Monte-Carlo false-positive rate

\approx 0.05

. Reproduced by script/power_analysis.py .

	Accuracy & scoring						Calibration
Forecaster	relWQL $\downarrow$	MASE $\downarrow$	MAPE $\downarrow$	sMAPE $\downarrow$	RMSE $\downarrow$	CRPS $\downarrow$	cov@80	bias(log)
seasonal_naive	1.000	0.787	0.295	0.263	35.8	14.54	0.86	$+0.027$
auto_ets	0.520	0.448	0.181	0.157	15.8	7.55	0.85	$+0.052$
auto_theta	0.528	0.463	0.181	0.160	15.9	7.67	0.89	$+0.045$
lightgbm_global	0.689	0.487	0.196	0.167	19.9	10.01	0.70	$+0.085$
chronos2	0.506	0.439	0.175	0.156	15.5	7.36	0.86	$+0.039$

Appendix

Table 5: Forecast-evaluation metrics on the held-out event. relWQL normalized to Seasonal-Naive (

=1.0

); for accuracy metrics

\downarrow

= better, best bold, second underlined; cov@80 and bias(log)

\overline{\log(\hat{y}/y)}

are calibration diagnostics (not ranked).

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,217 papers

TimeSeek: Temporal Reliability of Agentic Forecasters
Apr 5, 2026Hamza Mostafa, Om Shastri, Dennis LeeLLM Agent Reliability Forecasting Benchmarks
The Wharton School, University of Pennsylvania
LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications
Aug 24, 2026Xiaogang Xu, Jiaqi Tang, Jianmin Chen +12LLM Agents LLM-Based Time Series Forecasting
Zhejiang University, China · The Hong Kong University of Science and Technology, Hong Kong · Northwestern Polytechnical University, China +4
Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents
Sep 24, 2026Liqin Ye, Haorui Wang, Fardin Ahmed +8LLM Agent Evaluation Forecasting Benchmarks
Georgia Institute of Technology · Amazon · University of Florida

Apr 5, 2026J/K move · Enter open · S save

TimeSeek: Temporal Reliability of Agentic Forecasters

Hamza Mostafa, Om Shastri, Dennis Lee

The Wharton School, University of Pennsylvania

LLM Agent Reliability Forecasting Benchmarks Probabilistic Forecasting Prediction Markets

Open paper AskPDF arXiv

We introduce TimeSeek, a benchmark for studying how the reliability of agentic LLM forecasters changes over a prediction market's lifecycle. We evaluate 10 frontier models on 150 CFTC-regulated Kalshi binary markets at five temporal checkpoints, with and without web search, for 15,000 forecasts total. Models are most competitive early in a market's life and on high-uncertainty markets, but much less competitive near resolution and on strong-consensus markets. Web search improves pooled Brier Skill Score (BSS) for every model overall, yet hurts in 12% of model-checkpoint pairs, indicating that retrieval is helpful on average but not uniformly so. Simple two-model ensembles reduce error without surpassing the market overall. These descriptive results motivate time-aware evaluation and selective-deference policies rather than a single market snapshot or a uniform tool-use setting.

Abstract

LLM Agent Evaluation

Explore similar work

TimeSeek: Temporal Reliability of Agentic Forecasters

LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications

Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents