Game Arena: Strategic LLM Evaluation in Competitive Environments

Bovard Doerschuk-Tiberi; Yao Yan; Justin Chiu; Hann Wang; Timothy Chung; Martyna Plomecka; John Schultz; Jon Lipovetz

cs.AISep 25, 2026

Game Arena: Strategic LLM Evaluation in Competitive Environments

Authors: Bovard Doerschuk-Tiberi, Yao Yan, Justin Chiu, Hann Wang, Timothy Chung, Martyna Plomecka, John Schultz, Jon Lipovetz, Clayton Drazner, Yuchen Zhuang, Jaimie Hwang, Nate Keating, Riley Jones, Andrew Lee, Oran Kelly, Ian Gemp, Michael Aaron, Laurel Prince, Kate Larson, Jeff Moser, Harrison Jobe, Chad Woodford, Siqi Liu, Andrew Wang, Bo Chang, Christopher D'Mello, Diane Chaleff, Addison Howard, Johnny Yip, Chuck Sugnet, Antonio Gulli, Meghan O'Connell, Will Cukierski, Nenad Tomasev, Dima Yeroshenko, Kinjal Parekh, Roxanne Daniel, Marc Lanctot, Domino Weir, Elsa Dong, Daniel Hennes, Melissa Nalubwama, Robert Fraser, Ryan Trostle, Jun Peng, Tom Mason, Lloyd Hightower, Chiamaka Chukwuka, Yuexiang Zhai, Phoebe Kirk, Yi Su, Yuting Han, Jie Ren, Chris Prichard, Sahand Sharifzadeh, Karim Hakimzadeh, DJ Sterling, Meg Risdal, Kate Olszewska, Ya Xu, Orhan Firat, Minmin Chen

Abstract

We introduce Kaggle Game Arena, an open and ever-expanding platform to evaluate large language models (LLMs) through competitive games. Different from static benchmarks, game arena enables models to play head-to-head matchups in structured environments where the gameplay strength naturally increases as models evolve, preventing performance saturation. This technical report details the infrastructure behind Game Arena and describes the three pilot game environments: Chess, Poker, and Werewolf. These environments span perfect information, imperfect information, and multiplayer game settings, enabling a systematic study of models' strategic planning, adaptation, and robustness under uncertainty. For each game, we provide a detailed description of the environment, evaluation metrics, and results from running full competitions across models. Through robust infrastructure and large-scale ground-truth based evaluation, Game Arena ensures reproducibility, transparency and generalizability to new games and variants over time.

Figures & tables

Figure 2: Chess Text Benchmarks. Left: Pairwise win-rate heatmap across models. Right: Bootstrapped win-rate distribution (95% CI).

Figure 3: Analysis of Stockfish engine evaluations Left: average win probablity over the course of game(Chess Text benchmark). Right: Gemini 3 Pro preview win probability v.s. each opponent(Chess Text benchmark).

Figure 4: Cost–performance trade-offs across four Game Arena environments. Each panel displays model performance against inference cost (per turn for Chess and Poker; per game for Werewolf). The dashed blue curve denotes the Pareto frontier within that environment, consisting of non-dominated models (no alternative achieves higher performance at equal or lower cost). Orange points indicate models that are Pareto-efficient in at least one Game Arena environment (union across panels), highlighting globally competitive models while preserving environment-specific frontiers.

Player	Hands	BB/100	VPIP	Att To Steal (SB)	Call BB v SB	Fold BB v SB	3Bet BB v SB
GPT-5.2	180,000	46.56	96.42	91.75	57.04	7.59	35.37
o3	180,000	29.69	88.90	87.36	43.83	21.91	34.25
Grok 4	180,000	27.11	92.04	95.02	30.79	12.53	56.68
Claude Opus 4.5	179,936	17.62	82.92	87.71	64.91	22.79	12.30
Claude Sonnet 4.5	180,000	12.83	87.29	88.61	69.26	14.44	16.30
Gemini 3 Flash Prev.	180,000	5.49	69.08	72.39	49.84	34.18	15.98

Table 2: Aggregate poker statistics for all models, sorted by win rate (BB/100). VPIP = voluntarily put money in pot. Att To Steal = button open-raise frequency. Call/Fold/3Bet BB v SB = big blind response frequencies when facing a button raise.

Figure 5: Poker evaluation. Left: Head-to-head match win rates (row model vs. column model). Models are sorted alphabetically. Cell values represent the percentage of matches won by the row model. Blue shading indicates win rates above 50%; red indicates below 50%. Diagonal cells (self vs. self) are excluded. Right: Bootstrapped distributions of win rates (BB/100) for all ten models. Dashed red line indicates break-even (0 BB/100). Dashed black lines within each violin denote the quartiles and median. Models are ordered by mean BB/100 from bottom (lowest) to top (highest).

Figure 6: Werewolf leaderboard results evaluated by Game Theoretic Evaluation (GTE). (A) Role-specific contributions: The overall player skill (mean Net Rating) broken down into role-specific contributions. Error bars represent the 95% confidence interval of the total Net Rating, estimated from 1,000 bootstrap samples across 31,472 games. (B) Bootstrap distribution: Distributions of GTE Net Rating (95% CI), with dashed lines denoting quartiles. The vertical line at 0 indicates the break-even rating, and models are ordered by mean Net Rating.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark (metric)	Model	Score (CI)
Chess Text Input (Elo)	Gemini 3 Pro Preview	$1325\,(\,1217,\ 1428\,)$
	Gemini 3 Flash Preview	$1297\,(\,1205,\ 1405\,)$
	o3	$1009\,(\,928,\ 1107\,)$
	GPT-5.2	$933\,(\,849,\ 1018\,)$
	Grok 4	$773\,(\,701,\ 868\,)$
	Grok 4.1 Fast Reasoning	$632\,(\,556,\ 725\,)$

Appendix

Table 1: Leaderboard metrics with confidence intervals shown as endpoints (lower, upper). For Elo and BB/100, endpoints are computed as

(\text{score}-\Delta^{-},\ \text{score}+\Delta^{+})

Figure 1: Chess Opening Benchmarks. Left: Pairwise win-rate heatmap across models. Right: Bootstrapped win-rate distribution (95% CI) showing Gemini 3 variants leading in performance stability.

Figure 2: Analysis of Stockfish engine evaluations Left: average win probablity over the course of game(chess text benchmark). Right: Gemini 3 Pro preview win probability v.s. each opponent(chess text benchmark).

Figure 3: Analysis of rethink behavior in Chess Text tasks. Left: Percentage of rethinking chances used in the first half of the game. Right: Mean rethink count per turn, segmented by model performance.

Figure 4: Analysis of rethink behavior in Chess Opening tasks. Left: Percentage of rethinking chances used in the first half of the game. Right: Mean rethink count per turn, segmented by model performance.

Mechanic (Options)	1. Standard	2. No Doc Self	3. Fixed Order	4. Balanced
Discussion Order [fixed, random first, rotate]	Random first	Random first	Fixed	Rotate
Vote Order (Day & Night) [fixed, random first, rotate]	Random first	Random first	Fixed	Rotate
Tie Break (Day & Night) [random exile, no tie break]	Random exile	Random exile	No tie break	No tie break
Doctor Self Save [allowed, disabled]	Allowed	Disabled	Disabled	Disabled
Doctor Consecutive Save [allowed, disabled]	Allowed	Allowed	Allowed	Disabled
Villager Win Rate (95% CI)	0.734 $\pm$ 0.039	0.724 $\pm$ 0.039	0.606 $\pm$ 0.043	0.567 $\pm$ 0.043

Appendix

Table 2: Villager Win Rate by Game Mechanics Configurations (500 games/condition)

Figure 7: Tournament graph illustrating the pairwise win margins between the eight evaluated models. A directed edge from model

A\rightarrow B

indicates that

A

achieved more pairwise victories against

B

, with the edge label quantifying the net vote differential

\delta(A,B)

. The resulting graph contains no cycles, establishing a clean, unambiguous dominance hierarchy among the models.

	Elo Rating		Copeland Score		Ranked Pairs Rank
Gemini 3 Pro Preview	151	[144, 160]	7.0	[7.0, 7.0]	1	[1, 1]
Gemini 3 Flash Preview	124	[116, 132]	6.0	[5.0, 6.0]	2	[2, 3]
GPT-5.2	122	[115, 130]	5.0	[4.0, 6.0]	3	[2, 4]
Claude Opus 4.5	110	[103, 119]	4.0	[4.0, 5.0]	4	[3, 4]
Claude Sonnet 4.5	96	[89, 105]	3.0	[3.0, 3.0]	5	[5, 5]
Grok 4.1 Fast Reasoning	50	[43, 59]	2.0	[1.0, 2.0]	6	[6, 7]

Appendix

Table 3: Comparison of alternative ranking metrics from the pairwise win margin dataset across all 31472 Werewolf games. The models are identically ordered across all methods. 95% confidence intervals are obtained by showing the values of the

2.5^{th}

and

97.5^{th}

percentiles.

Figure 8: Heatmaps for cooperative game-theoretic analysis showing the substitution analysis across all models, for the roles of Villager , Werewolf , Seer , and Doctor . The $x$ -axes represent the models that are being substituted into the team whereas the $y$ -axis represents the models that are being replaced. We can quickly observe that it is always beneficial to replace any model with Gemini 3 Pro Preview, on any team, whereas for a model like Claude Sonnet 4.5, the benefits are more nuanced since the value of the substitution depends on both the coalition and the model being replaced.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,217 papers

LM Fight Arena: Benchmarking Large Multimodal Models via Game Competition
Oct 10, 2025Yushuo Zheng, Tongrui Ye, Zicheng Zhang +3Multimodal Large Language Models AI Agent Benchmarks
Shanghai Jiao Tong University, Shanghai, China · Shanghai AI Lab, Shanghai, China · Tongji University, Shanghai, China
A Unified Arena for LLM Agents across Diverse Eurogame
Oct 26, 2026Fengrui Zhang, Xiang Li, Ruomeng Ding +6LLM Agent Evaluation Game-Playing Agents
Alibaba Group · Nanjing University · National University of Singapore +3
GENSTRAT: Toward a Science of Strategic Reasoning in Large Language Models
May 22, 2026Vartan Shadarevian, Kia Ghods, Alex Kenich +1LLM Evaluation Imperfect-Information Games
Princeton University · Google

Oct 10, 2025J/K move · Enter open · S save

LM Fight Arena: Benchmarking Large Multimodal Models via Game Competition

Yushuo Zheng, Tongrui Ye, Zicheng Zhang, Xiongkuo Min+2

Shanghai Jiao Tong University, Shanghai, China · Shanghai AI Lab, Shanghai, China · Tongji University, Shanghai, China

Multimodal Large Language Models AI Agent Benchmarks Multimodal Model Evaluation

Open paper AskPDF arXiv

Existing benchmarks for large multimodal models (LMMs) often fail to capture their performance in real-time, adversarial environments. We introduce LM Fight Arena (Large Model Fight Arena), a novel framework that evaluates LMMs by pitting them against each other in the classic fighting game Mortal Kombat II, a task requiring rapid visual understanding and tactical, sequential decision-making. In a controlled tournament, we test six leading open- and closed-source models, where each agent operates controlling the same character to ensure a fair comparison. The models are prompted to interpret game frames and state data to select their next actions. Unlike static evaluations, LM Fight Arena provides a fully automated, reproducible, and objective assessment of an LMM's strategic reasoning capabilities in a dynamic setting. This work introduces a challenging and engaging benchmark that bridges the gap between AI evaluation and interactive entertainment.

Abstract

LLM Agent Evaluation

Explore similar work

LM Fight Arena: Benchmarking Large Multimodal Models via Game Competition

A Unified Arena for LLM Agents across Diverse Eurogame

GENSTRAT: Toward a Science of Strategic Reasoning in Large Language Models