Vendster: An Evaluation Arena to Investigate How Forecasting Tool Quality Affects LLM Agent Behaviors in Buyer-Vendor Negotiation

Charin Polpanumas

cs.GTOct 28, 2026

Vendster: An Evaluation Arena to Investigate How Forecasting Tool Quality Affects LLM Agent Behaviors in Buyer-Vendor Negotiation

Authors: Charin Polpanumas

Organizations: Amazon, Japan

Abstract

Time-series forecasters are ranked almost entirely by accuracy metrics, but a forecast's decision value, its effect on the downstream decision, need not track accuracy and is rarely measured. This problem compounds as large language model (LLM) agents increasingly consume forecasters as tools, mediating them through stochastic, opaque policies. We present Vendster, an evaluation arena measuring how a forecaster's bias and prediction-interval width change the outcome of a buyer--vendor discount negotiation between two LLM agents. In the reference instance Vendster-M5, sampling base demand patterns from the M5 dataset, we evaluate two frontier-closed LLM families (Claude Opus 4.6 and GPT-5.6 Sol) and two open ones (DeepSeek-V3.2 and Qwen3-235B-A22B). Model-generated forecasts raise sales target attainment by up to +17 percentage points (p<0.001), yet standard forecast evaluation metrics such as relWQL and MASE do not predict negotiation outcomes at a statistically significant level. To isolate the causal structure, we substitute in synthetic forecasts with controlled bias and interval width. Under-prediction bias significantly improves outcomes for agents that react more to the forecasts---dose-response slopes -0.387 and -0.416 (p<0.001). Interval width has no statistically significant effect at zero bias. These findings indicate that the direction of forecast bias and agent forecast-reactivity are more strongly associated with downstream outcomes than forecast accuracy. Vendster is fully customizable and released under CC BY-NC 4.0 at https://github.com/amazon-science/Vendster.

Figures & tables

			Causal		Observational
LLM	No Forecast	Forecast Uplift	Bias	Width	Bias	Width	relWQL
Opus	0.24	$+0.12^{*}$ (.000, .000)	$-0.387^{*}$ (.000, .000)	$+0.016$ (.225, .899)	$-0.466^{*}$ (.009, .034)	$+0.000$ (.904, 1.000)	$-0.021$ (.703, 1.000)
GPT	0.25	$+0.17^{*}$ (.000, .000)	$-0.416^{*}$ (.000, .000)	$+0.002$ (.882, 1.000)	$-0.408$ (.057, .171)	$-0.002$ (.053, .211)	$-0.126$ (.016, .062)
Qwen	0.11	$+0.11^{*}$ (.000, .000)	$+0.001$ (.988, 1.000)	$-0.003$ (.796, 1.000)	$-0.195$ (.266, .510)	$+0.001$ (.423, 1.000)	$+0.009$ (.835, 1.000)
DeepSeek	0.22	$+0.08^{*}$ (.001, .001)	$+0.012$ (.855, 1.000)	$-0.002$ (.867, 1.000)	$-0.182$ (.255, .510)	$+0.001$ (.574, 1.000)	$+0.044$ (.362, 1.000)

Table 1: Forecast effects by LLM. Forecast Uplift is the H0 uplift in sales target attainment rate from model forecasts over Buyer agent with no forecast. Causal columns are slopes on the varied bias/width grid; Observational columns regress attainment on measured bias, width, and relWQL. Cells show effect

(\text{raw}\ p,\ \text{Holm}\ p)

; ∗ : Holm

p<0.05

across the four LLMs.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

metric	Opus	GPT	Qwen	DeepSeek
relWQL	$-0.021$ (.703, 1.000)	$-0.126$ (.016, .062)	$+0.009$ (.835, 1.000)	$+0.044$ (.362, 1.000)
MASE	$-0.095$ (.362, 1.000)	$-0.194$ (.104, .415)	$+0.005$ (.961, 1.000)	$+0.055$ (.510, 1.000)
MAPE	$-0.156$ (.535, 1.000)	$-0.353$ (.261, .783)	$+0.015$ (.953, 1.000)	$+0.308$ (.154, .616)
sMAPE	$-0.195$ (.598, 1.000)	$-0.408$ (.424, 1.000)	$+0.098$ (.765, 1.000)	$+0.387$ (.147, .590)
RMSE	$-0.003$ (.453, 1.000)	$-0.007$ (.122, .488)	$+0.001$ (.854, 1.000)	$+0.003$ (.488, 1.000)
CRPS	$-0.004$ (.477, 1.000)	$-0.011$ (.068, .270)	$+0.003$ (.625, 1.000)	$+0.006$ (.363, 1.000)

Appendix

Table 2: Forecast quality

\to

sales target attainment: per-basket slope by metric

\times

model, cell

=

slope

(\text{raw}\ p,\ \text{Holm}\ p)

; ∗ : Holm

p<0.05

		first-offer	agreed disc.	concession	turns	agreement
Opus	presence	$+0.046^{*}$ (.000, .000)	$+0.030^{*}$ (.000, .000)	$+0.005^{*}$ (.000, .000)	$+1.03^{*}$ (.000, .000)	$+0.003$ (.333, .429)
Opus	bias slope	$-0.101^{*}$ (.000, .000)	$-0.082^{*}$ (.000, .000)	$+0.002$ (.341, 1.000)	$-3.09^{*}$ (.000, .000)	$+0.002$ (.544, .928)
GPT	presence	$+0.020^{*}$ (.000, .000)	$+0.053^{*}$ (.000, .000)	$-0.016^{*}$ (.000, .000)	$+6.61^{*}$ (.000, .000)	$-0.135^{*}$ (.000, .000)
GPT	bias slope	$-0.139^{*}$ (.000, .000)	$-0.195^{*}$ (.000, .000)	$+0.047^{*}$ (.000, .000)	$-21.94^{*}$ (.000, .000)	$+0.926^{*}$ (.000, .000)
Qwen	presence	$+0.105^{*}$ (.000, .000)	$+0.041^{*}$ (.000, .000)	$-0.003$ (.168, .168)	$-1.68^{*}$ (.000, .000)	$+0.155^{*}$ (.000, .000)
Qwen	bias slope	$+0.026^{*}$ (.003, .007)	$-0.004$ (.577, 1.000)	$-0.001$ (.639, 1.000)	$+0.08$ (.906, .906)	$+0.038$ (.464, .928)

Appendix

Table 3: Forecast-reactivity behaviors. Columns are the logged negotiation behaviors: first-offer (mean discount in the Buyer’s opening proposal), agreed disc. (mean discount at agreement), concession (mean absolute change between a proposer’s successive proposal means), turns (turns to termination), and agreement (deal rate). For each family, two rows: presence —a behavior’s uplift from having a forecast—and bias slope —its per-basket slope along the varied bias ladder. Cells show the statistic over

(\text{raw}\ p,\ \text{Holm}\ p)

; ∗ : Holm

p<0.05

Figure 1: Sales target attainment against synthetically varied bias (top) and interval width (bottom), one column per model, with 95% confidence intervals.

Figure 2: Bias-attainment slope by using different LLMs for Buyers and Vendors; this suggests the slope is Buyer-dependent.

Figure 3: Operating-point selection using the no-forecast Sonnet 4.6 agent, , with 95% confidence intervals.

Hypothesis	effect tested	power ( $\Delta{=}0.10$ )	floor (attain)
H0	aggregate forecast vs no_forecast	0.81	—
H1c	bias, causal grid	0.98	0.070
H2c	width, causal grid	0.94	0.080
H1o	bias, observational	0.88	0.089
H2o	width, observational	0.97	0.072
H3o	quality (relWQL), observational	0.97	0.074

Appendix

Table 4: A-priori power at the locked design (

K{=}16

m{=}25

\alpha{=}0.05

) for the

0.10

-attainment MDE. Power : probability of detecting a

0.10

effect; floor : minimum detectable effect at

80\%

power. H0 is sized directly to the MDE; slope power is range-independent, so the observational tests size despite the forecasters’ tight spread. Monte-Carlo false-positive rate

\approx 0.05

. Reproduced by script/power_analysis.py .

	Accuracy & scoring						Calibration
Forecaster	relWQL $\downarrow$	MASE $\downarrow$	MAPE $\downarrow$	sMAPE $\downarrow$	RMSE $\downarrow$	CRPS $\downarrow$	cov@80	bias(log)
seasonal_naive	1.000	0.787	0.295	0.263	35.8	14.54	0.86	$+0.027$
auto_ets	0.520	0.448	0.181	0.157	15.8	7.55	0.85	$+0.052$
auto_theta	0.528	0.463	0.181	0.160	15.9	7.67	0.89	$+0.045$
lightgbm_global	0.689	0.487	0.196	0.167	19.9	10.01	0.70	$+0.085$
chronos2	0.506	0.439	0.175	0.156	15.5	7.36	0.86	$+0.039$

Appendix

Table 5: Forecast-evaluation metrics on the held-out event. relWQL normalized to Seasonal-Naive (

=1.0

); for accuracy metrics

\downarrow

= better, best bold, second underlined; cov@80 and bias(log)

\overline{\log(\hat{y}/y)}

are calibration diagnostics (not ranked).

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,217 papers

Apr 5, 2026

TimeSeek: Temporal Reliability of Agentic Forecasters

We introduce TimeSeek, a benchmark for studying how the reliability of agentic LLM forecasters changes over a prediction market's lifecycle. We evaluate 10 frontier models on 150 CFTC-regulated Kalshi binary markets at five temporal checkpoints, with and without web search, for 15,000 forecasts total. Models are most competitive early in a market's life and on high-uncertainty markets, but much less competitive near resolution and on strong-consensus markets. Web search improves pooled Brier Skill Score (BSS) for every model overall, yet hurts in 12% of model-checkpoint pairs, indicating that retrieval is helpful on average but not uniformly so. Simple two-model ensembles reduce error without surpassing the market overall. These descriptive results motivate time-aware evaluation and selective-deference policies rather than a single market snapshot or a uniform tool-use setting.

Hamza Mostafa, Om Shastri, Dennis Lee

The Wharton School, University of Pennsylvania

LLM Agent Reliability Forecasting Benchmarks Probabilistic Forecasting Prediction Markets

Ask PDF

Aug 24, 2026

LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications

Large language models (LLMs) now support forecasting systems that combine language-based reasoning with temporal data, evidence retrieval, external tools, and iterative prediction. We investigate LLM-based forecasting agents, meaning systems in which a language model contributes to a scored prediction about a future or currently unobserved target. We organize architectures into three groups. Standalone LLM workflows operate on encoded time series or event context. Tool- and retrieval-augmented agents incorporate external evidence. Hybrid systems pair LLMs with statistical or foundation models. We then review training methods and evaluation protocols. We examine negative as well as positive evidence, including sensitivity to small input perturbations, ablations in which the LLM component does not improve accuracy, and benchmark gains that may reflect contamination instead of temporal reasoning. We cover applications in finance, weather, health, energy, and operations, and we summarize the benchmarks and datasets used for evaluation. The evidence indicates that measurement is a central limitation. Future work requires calibration under distribution shift, contamination-resistant live evaluation, explicit reporting of cost and accuracy together, and methods for handling feedback between deployed forecasts and the outcomes being forecast.

Xiaogang Xu, Jiaqi Tang, Jianmin Chen +12

Zhejiang University, China · The Hong Kong University of Science and Technology, Hong Kong · Northwestern Polytechnical University, China +4

LLM Agents LLM-Based Time Series Forecasting LLM Agent Evaluation LLM Agent Training Forecasting Benchmarks

Ask PDF

Sep 24, 2026

Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents

Forecasting requires large language model (LLM) agents to gather evolving evidence, revise their beliefs, and make calibrated predictions over time. Yet evaluating and improving these capabilities is challenging because real-world events unfold continuously and outcomes may take months to resolve. We introduce Forecast-Dojo, a replayable environment that reconstructs resolved prediction-market events as sequences of dated forecasting steps, enabling reproducible evaluation, longitudinal analysis, and training from agent interactions. It comprises 1,568 Polymarket events split temporally into training and evaluation sets, alongside 18.8 million dated news articles. Across 12 models, research tools consistently improve forecasting performance, and greater capture of new evidence is associated with larger improvements over time. Nevertheless, all models trail historical market forecasts, while carrying a belief notebook across dates reduces research costs without consistently improving forecast quality. We supervise fine-tuned Qwen3-30B-A3B-Thinking-2507 on collected trajectories from Qwen3-235B-A22B-Thinking-2507 to demonstrate the environment's training utility. The student's accuracy is improved from 34.8% to 42.8% on the evaluation set.

Liqin Ye, Haorui Wang, Fardin Ahmed +8

Georgia Institute of Technology · Amazon · University of Florida

LLM Agent Evaluation Forecasting Benchmarks LLM Agent Training

Ask PDF

Abstract

LLM Agent Evaluation

Explore similar work

TimeSeek: Temporal Reliability of Agentic Forecasters

LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications

Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents