[Re] Benchmarking LLM Capabilities in Negotiation through Scoreable Games

Jorge Carrasco Pollo; Ioannis Kapetangeorgis; Joshua Rosenthal; John Hua Yao

cs.LGFeb 20, 2026

[Re] Benchmarking LLM Capabilities in Negotiation through Scoreable Games

Authors: Jorge Carrasco Pollo, Ioannis Kapetangeorgis, Joshua Rosenthal, John Hua Yao

Organizations: Informatics Institute University of Amsterdam

Abstract

Large Language Models (LLMs) demonstrate significant potential in multi-agent negotiation tasks, yet evaluation in this domain remains challenging due to a lack of robust and generalizable benchmarks. Abdelnabi et al. (2024) introduce a negotiation benchmark based on Scoreable Games, with the aim of developing a highly complex and realistic evaluation framework for LLMs. Our work investigates the reproducibility of claims in their benchmark, and provides a deeper understanding of its usability and generalizability. We replicate the original experiments on additional models, and introduce additional metrics to verify negotiation quality and evenness of evaluation. Our findings reveal that while the benchmark is indeed complex, model comparison is ambiguous, raising questions about its objectivity. Furthermore, we identify limitations in the experimental setup, particularly in information leakage detection and thoroughness of the ablation study. By examining and analyzing the behavior of a wider range of models on an extended version of the benchmark, we reveal insights that provide additional context to potential users. Our results highlight the importance of context in model-comparative evaluations.

Figures & tables

	Model	Final $\uparrow$		Any $\uparrow$	Wrong $\downarrow$	Leaked $\downarrow$
		5-way	6-way
( 1 )	GPT-4o mini	55	5	90	2.69	0.58
( , 2024 )	GPT-4o	75	10	85	0.58	0.00
( Touvron et al., 2023 )	Llama-2-13b (int8)	30	0	75	17.92	9.23
( Grattafiori et al., 2024 )	Llama-3-8B	25	0	70	8.14	69.42
( Grattafiori et al., 2024 )	Llama-3.3-70B (int4)	60	0	100	0.96	0.00

Table 1: Performance of models on the base game measured in %. Bolded model names indicate common models, quantized, with the original study. The results of models suffixed by “*” are taken from the original study ( Abdelnabi et al., 2024 ) .

Ablations (✓= No Ablation, ✗= Ablated)				GPT-4o mini/Qwen-72B
Prev. deals	Others. Prefer.	Candidates	Planning	5/6-way	6-way	Any
✓	✓	✓	✓	60 / 90	0 / 0	100 / 100
✓	✓	✓	✗	55 / 95	10 / 0	95 / 95
✓	✓	✗	✓	60 / 90	0 / 0	85 / 90
✓	✓	✗	✗	45 / 80	10 / 0	95 / 100
✓	✗	✓	✓	5 / 90	0 / 0	10 / 90

Table 2: Ablation study results. The ablation configuration used in original work for GPT-4 is highlighted in red. Large variance in performance is visible across configurations.

	GPT4 ( Abdelnabi et al., 2024 )			GPT-4o mini			Mistral-Small			Qwen2.5-72B
	5/6-way	6-way	Any%	5/6-way	6-way	Any%	5/6-way	6-way	Any%	5/6-way	6-way	Any%
Base	81	33	100	55	5	90	80	0	100	85	0	95
Base_rewritten	86	24	100	55	0	80	60	15	95	95	5	100
Game 1	65	10	85	35	5	75	25	5	75	80	30	95
Game 2	70	40	90	15	15	30	40	40	85	30	15	60
Game 3	86	81	95	55	55	95	0	0	50	80	80	100

Table 3: Performance of models across different games.

Panel of Figure 1 — Figure 1: Evolution in different USW (blue), ESW (green), and NSW (red) metrics over time across two different models. The solid lines represent the mean score over time. The dashed colored lines represent the minimum and maximum achievable scores under a given metric. The black dashed line over USW is the linear least square regression.

	${\|\Pi_{\text{acc}}\|}/{\|\Pi\|}$	${\|\Pi_{\text{hard}}\|}/{\|\Pi\|}$	% Spar.	% IoU
Base	${55}/{720}$	${12}/{720}$	38.6	18.75
Base_rewritten	${55}/{720}$	${12}/{720}$	38.6	18.75
Game 1	${57}/{720}$	${21}/{720}$	23.68	29.77
Game 2	${57}/{720}$	${18}/{720}$	29.82	25.75
Game 3	${55}/{720}$	${35}/{720}$	42.98	21.65

Table 5: Score function statistics.

Panel of Figure 2 — Figure 2: Varying thresholds and number of players in base game for GPT-4o mini

Variant	GPT-4o mini		Qwen2.5-72B		GPT4 *
Variant	5/6-way	6-way	5/6-way	6-way	5/6-way	6-way
All compromising	55	5	85	0	81	33
One greedy ( $p_{i}\in P_{\text{const}}$ )	80	10	95	0	57	30
One greedy ( $p_{1}$ )	45	5	50	5	27	9
Two greedy ( $P_{\text{benefit}}$ )	95	5	75	0	65	15
All greedy	50	0	15	5	26	11

Table 6: Performance in the different behavioral variants, comparing GPT-4o mini, Qwen2.5-72B, and GPT4.

P_{\text{benefit}}

is the set of players whose goals align to those of

p_{1}

and

P_{\text{const}}

is similarly defined but with neutral goals with respect to

p_{1}

. It is not clear how these sets are delineated in the original paper, so we replicated the setup with the players indicated in the open source configurations. The results for GPT4, marked with *, were taken from the original work.

Panel of Figure 3 — Figure 3: Evolution of deals for several behavioral configurations. Environmental League is $p_{i}$ and acts as greedy for the greedy experiment and as adversarial in both the untargeted and targeted experiments. For the latter, Local Labour Union is the target.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Parameters (B)	Architecture	Context Window	Additional Details
Llama-3.3-70B	70	Decoder-only Transformer	128k tokens	SwiGLU, RoPE, RMSNorm
Mistral-Small	22	Cost-effective Transformer	33k tokens	Task-optimized
Phi-3.5-mini	3.8	Compact Transformer	4k tokens	Efficiency-focused
Mistral-8B	8	Performance-optimized	128k tokens	Balanced efficiency
Qwen2.5-72B	72	Dense decoder-only	131k tokens	Advanced capabilities
Phi-4	14	State-of-the-art Transformer	16k tokens	High performance

Appendix

Table 7: Summary of open-source models evaluated, including their parameters, architecture, context window size, and additional details.

Model	Quantization Applied
Llama-2-13B	int8
Llama-3.3-70B	int4
Qwen2.5-72B	int4
Phi-4	int8
Mistral-Small	int8
Mixtral-8x7B	int4

Appendix

Table 8: Quantized models and techniques applied

Model	Runtime (min)
GPT-4o mini	4
Mistral-Small-Instruct-2409	20
Phi-3.5-mini-instruct	6
Llama-3.3-70B-Instruct	22
Ministral-8B-Instruct-2410	15
Phi-4	15

Appendix

Table 9: Average runtime per negotiation session (24 rounds) across different models.

Model	Average Emissions (kg CO 2 e)
Llama-2-13B	0.4
Llama-3.3-70B	1.0
Meta-Llama-3-8B	0.1
Ministral-8B	0.6
Mistral-Small	0.7
Mixtral_8x7B	0.3

Appendix

Table 10: Estimated average carbon emissions (

kg\ \ CO_{2}e

) per model for processing a batch of 20 games

Model	Original Code					Our Code
	Final ↑ (%)		Any ↑ (%)	Wrong ↓ (%)	Leaked ↓ (%)	Final ↑ (%)		Any ↑ (%)	Wrong ↓ (%)	Leaked ↓ (%)	Failed ↓ (%)
	5-way	6-way				5-way	6-way
Llama-2-13b	30.0	0.0	75.0	17.92	9.23	8.33	0.0	25.0	25.51	0.32	40
Llama-3-8B	25.0	0.0	70.0	8.14	69.42	35.0	0.0	55.0	9.8	0	0
Ministral-8B	25.0	0.0	50.0	12.88	7.12	22.22	0.0	50.0	11.11	0	10
Mixtral-8x7B	30.0	5.0	55.0	23.64	24.81	26.67	0.0	66.67	20.14	0	25

Appendix

Table 11: Comparison of model performance metrics before and after the leakage issue is resolved.

	Original Code			Our code
Model	Successful Games	Leaked Games	Leaked (%)	Successful Games	Leaked Games	Leaked (%)
Llama-2-13b	20	11	9.23	12	1	0.32
Llama-3-8B	20	20	69.42	20	0	0.00
Ministral-8B	20	20	7.12	18	0	0.00
Mixtral-8x7B	20	20	24.81	15	0	0.00
Phi-3.5-mini	20	20	40.77	20	2	0.38

Appendix

Table 12: A comparison of models that are prone to leakage before and after code adjustments.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

TERMS-Bench: Diagnosing LLM Negotiation Agents Beyond Deal Rate
May 13, 2026Erica Zhang, Fangzhao Zhang, Aneesh Pappu +5Multi-Agent Negotiation Automated Negotiation
♠Stanford School of Engineering · ♣Stanford Department of Economics · ♥Stanford Graduate School of Business +1
The Language of Bargaining: Linguistic Effects in LLM Negotiations
Jan 7, 2026Stuti Sinha, Himanshu Kumar, Aryan Raju Mandapati +2Multilingual Language Model Evaluation Multi-Agent Negotiation
BITS Pilani
Cattle Trade: A Multi-Agent Benchmark for LLM Bluffing, Bidding, and Bargaining
May 14, 2026Robert Müller, Clemens MüllerMulti-Agent Negotiation Game-Playing Agents

May 13, 2026J/K move · Enter open · S save

TERMS-Bench: Diagnosing LLM Negotiation Agents Beyond Deal Rate

Erica Zhang, Fangzhao Zhang, Aneesh Pappu, Batu El+4

♠Stanford School of Engineering · ♣Stanford Department of Economics · ♥Stanford Graduate School of Business+1

Multi-Agent Negotiation Automated Negotiation Imperfect-Information Games Game Theory LLM Agent Evaluation

Open paper AskPDF arXiv

Negotiation is a central mechanism of economic exchange, shaping markets, procurement, labor agreements, and resource allocation. It is also a canonical testbed for agentic language models, requiring multi-turn interaction under hidden preferences, strategic communication, and binding constraints. These properties make negotiation hard to evaluate: unlike math or code, it has no intrinsic verifier. Existing LLM negotiation evaluations rely on LLM-vs.-LLM interaction or aggregate outcomes such as deal rate, leaving failures opaque. We introduce Terms-Bench, short for Testbed for Economic Reasoning in Multi-turn Strategy, a Bayesian-game framework that makes the environment itself the verifier by specifying the counterpart's latent type, policy, and payoff structure. We instantiate it in bilateral price negotiation, where the counterpart's private state and simulator policy are hidden from the agent but observable to the evaluator. This turns the counterpart from a black-box opponent into a diagnostic instrument, enabling agent-attributable failure analysis and oracle-reference optimality gaps. Evaluating 13 LLM agents spanning frontier systems from major providers, Terms-Bench turns negotiation evaluation from aggregate ranking into actionable diagnosis: where agents fail, why they fail, and what to strengthen. Empirically, frontier models saturate deal rate yet diverge in surplus extraction, cue use, belief calibration, and compliance, revealing agent-specific bargaining bottlenecks masked by prior benchmarks.

Abstract

LLM Evaluation

Explore similar work

TERMS-Bench: Diagnosing LLM Negotiation Agents Beyond Deal Rate

The Language of Bargaining: Linguistic Effects in LLM Negotiations

Cattle Trade: A Multi-Agent Benchmark for LLM Bluffing, Bidding, and Bargaining