Game Arena: Strategic LLM Evaluation in Competitive Environments
Authors: Bovard Doerschuk-Tiberi, Yao Yan, Justin Chiu, Hann Wang, Timothy Chung, Martyna Plomecka, John Schultz, Jon Lipovetz, Clayton Drazner, Yuchen Zhuang, Jaimie Hwang, Nate Keating, Riley Jones, Andrew Lee, Oran Kelly, Ian Gemp, Michael Aaron, Laurel Prince, Kate Larson, Jeff Moser, Harrison Jobe, Chad Woodford, Siqi Liu, Andrew Wang, Bo Chang, Christopher D'Mello, Diane Chaleff, Addison Howard, Johnny Yip, Chuck Sugnet, Antonio Gulli, Meghan O'Connell, Will Cukierski, Nenad Tomasev, Dima Yeroshenko, Kinjal Parekh, Roxanne Daniel, Marc Lanctot, Domino Weir, Elsa Dong, Daniel Hennes, Melissa Nalubwama, Robert Fraser, Ryan Trostle, Jun Peng, Tom Mason, Lloyd Hightower, Chiamaka Chukwuka, Yuexiang Zhai, Phoebe Kirk, Yi Su, Yuting Han, Jie Ren, Chris Prichard, Sahand Sharifzadeh, Karim Hakimzadeh, DJ Sterling, Meg Risdal, Kate Olszewska, Ya Xu, Orhan Firat, Minmin Chen
We introduce Kaggle Game Arena, an open and ever-expanding platform to evaluate large language models (LLMs) through competitive games. Different from static benchmarks, game arena enables models to play head-to-head matchups in structured environments where the gameplay strength naturally increases as models evolve, preventing performance saturation. This technical report details the infrastructure behind Game Arena and describes the three pilot game environments: Chess, Poker, and Werewolf. These environments span perfect information, imperfect information, and multiplayer game settings, enabling a systematic study of models' strategic planning, adaptation, and robustness under uncertainty. For each game, we provide a detailed description of the environment, evaluation metrics, and results from running full competitions across models. Through robust infrastructure and large-scale ground-truth based evaluation, Game Arena ensures reproducibility, transparency and generalizability to new games and variants over time.
Figures & tables
Figure 1: Game Arena infrastructure.
Figure 2: Chess Text Benchmarks. Left: Pairwise win-rate heatmap across models. Right: Bootstrapped win-rate distribution (95% CI).
Figure 3: Analysis of Stockfish engine evaluations Left: average win probablity over the course of game(Chess Text benchmark). Right: Gemini 3 Pro preview win probability v.s. each opponent(Chess Text benchmark).
Figure 4: Cost–performance trade-offs across four Game Arena environments. Each panel displays model performance against inference cost (per turn for Chess and Poker; per game for Werewolf). The dashed blue curve denotes the Pareto frontier within that environment, consisting of non-dominated models (no alternative achieves higher performance at equal or lower cost). Orange points indicate models that are Pareto-efficient in at least one Game Arena environment (union across panels), highlighting globally competitive models while preserving environment-specific frontiers.
Player
Hands
BB/100
VPIP
Att To Steal (SB)
Call BB v SB
Fold BB v SB
3Bet BB v SB
GPT-5.2
180,000
46.56
96.42
91.75
57.04
7.59
35.37
o3
180,000
29.69
88.90
87.36
43.83
21.91
34.25
Grok 4
180,000
27.11
92.04
95.02
30.79
12.53
56.68
Claude Opus 4.5
179,936
17.62
82.92
87.71
64.91
22.79
12.30
Claude Sonnet 4.5
180,000
12.83
87.29
88.61
69.26
14.44
16.30
Gemini 3 Flash Prev.
180,000
5.49
69.08
72.39
49.84
34.18
15.98
Table 2: Aggregate poker statistics for all models, sorted by win rate (BB/100). VPIP = voluntarily put money in pot. Att To Steal = button open-raise frequency. Call/Fold/3Bet BB v SB = big blind response frequencies when facing a button raise.
Figure 5: Poker evaluation. Left: Head-to-head match win rates (row model vs. column model). Models are sorted alphabetically. Cell values represent the percentage of matches won by the row model. Blue shading indicates win rates above 50%; red indicates below 50%. Diagonal cells (self vs. self) are excluded. Right: Bootstrapped distributions of win rates (BB/100) for all ten models. Dashed red line indicates break-even (0 BB/100). Dashed black lines within each violin denote the quartiles and median. Models are ordered by mean BB/100 from bottom (lowest) to top (highest).
Figure 6: Werewolf leaderboard results evaluated by Game Theoretic Evaluation (GTE). (A) Role-specific contributions: The overall player skill (mean Net Rating) broken down into role-specific contributions. Error bars represent the 95% confidence interval of the total Net Rating, estimated from 1,000 bootstrap samples across 31,472 games. (B) Bootstrap distribution: Distributions of GTE Net Rating (95% CI), with dashed lines denoting quartiles. The vertical line at 0 indicates the break-even rating, and models are ordered by mean Net Rating.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark (metric)
Model
Score (CI)
Chess Text Input (Elo)
Gemini 3 Pro Preview
1325(1217,1428)
Gemini 3 Flash Preview
1297(1205,1405)
o3
1009(928,1107)
GPT-5.2
933(849,1018)
Grok 4
773(701,868)
Grok 4.1 Fast Reasoning
632(556,725)
Appendix
Table 1: Leaderboard metrics with confidence intervals shown as endpoints (lower, upper). For Elo and BB/100, endpoints are computed as (score−Δ−,score+Δ+) .
Figure 1: Chess Opening Benchmarks. Left: Pairwise win-rate heatmap across models. Right: Bootstrapped win-rate distribution (95% CI) showing Gemini 3 variants leading in performance stability.
Figure 2: Analysis of Stockfish engine evaluations Left: average win probablity over the course of game(chess text benchmark). Right: Gemini 3 Pro preview win probability v.s. each opponent(chess text benchmark).
Figure 3: Analysis of rethink behavior in Chess Text tasks. Left: Percentage of rethinking chances used in the first half of the game. Right: Mean rethink count per turn, segmented by model performance.
Figure 4: Analysis of rethink behavior in Chess Opening tasks. Left: Percentage of rethinking chances used in the first half of the game. Right: Mean rethink count per turn, segmented by model performance.
Mechanic (Options)
1. Standard
2. No Doc Self
3. Fixed Order
4. Balanced
Discussion Order [fixed, random first, rotate]
Random first
Random first
Fixed
Rotate
Vote Order (Day & Night) [fixed, random first, rotate]
Random first
Random first
Fixed
Rotate
Tie Break (Day & Night) [random exile, no tie break]
Random exile
Random exile
No tie break
No tie break
Doctor Self Save [allowed, disabled]
Allowed
Disabled
Disabled
Disabled
Doctor Consecutive Save [allowed, disabled]
Allowed
Allowed
Allowed
Disabled
Villager Win Rate (95% CI)
0.734 ± 0.039
0.724 ± 0.039
0.606 ± 0.043
0.567 ± 0.043
Appendix
Table 2: Villager Win Rate by Game Mechanics Configurations (500 games/condition)
Figure 6: Ratings of group Elo, OpenSkill, and Game Theoretic Eval (GTE). The means and 95% confidence intervals were derived from 1,000 bootstrap iterations across the 31,472 match-ups. Even though the relative rankings and trends stay consistent across the three methods, Game Theoretic Evaluation exhibits tighter confidence intervals and greater discriminative power, whereas group Elo and OpenSkill struggle to distinctly separate the top-performing models.
Figure 7: Tournament graph illustrating the pairwise win margins between the eight evaluated models. A directed edge from model A→B indicates that A achieved more pairwise victories against B , with the edge label quantifying the net vote differential δ(A,B) . The resulting graph contains no cycles, establishing a clean, unambiguous dominance hierarchy among the models.
Elo Rating
Copeland Score
Ranked Pairs Rank
Gemini 3 Pro Preview
151
[144, 160]
7.0
[7.0, 7.0]
1
[1, 1]
Gemini 3 Flash Preview
124
[116, 132]
6.0
[5.0, 6.0]
2
[2, 3]
GPT-5.2
122
[115, 130]
5.0
[4.0, 6.0]
3
[2, 4]
Claude Opus 4.5
110
[103, 119]
4.0
[4.0, 5.0]
4
[3, 4]
Claude Sonnet 4.5
96
[89, 105]
3.0
[3.0, 3.0]
5
[5, 5]
Grok 4.1 Fast Reasoning
50
[43, 59]
2.0
[1.0, 2.0]
6
[6, 7]
Appendix
Table 3: Comparison of alternative ranking metrics from the pairwise win margin dataset across all 31472 Werewolf games. The models are identically ordered across all methods. 95% confidence intervals are obtained by showing the values of the 2.5th and 97.5th percentiles.
Figure 8: Heatmaps for cooperative game-theoretic analysis showing the substitution analysis across all models, for the roles of Villager , Werewolf , Seer , and Doctor . The x -axes represent the models that are being substituted into the team whereas the y -axis represents the models that are being replaced. We can quickly observe that it is always beneficial to replace any model with Gemini 3 Pro Preview, on any team, whereas for a model like Claude Sonnet 4.5, the benefits are more nuanced since the value of the substitution depends on both the coalition and the model being replaced.
Existing benchmarks for large multimodal models (LMMs) often fail to capture their performance in real-time, adversarial environments. We introduce LM Fight Arena (Large Model Fight Arena), a novel framework that evaluates LMMs by pitting them against each other in the classic fighting game Mortal Kombat II, a task requiring rapid visual understanding and tactical, sequential decision-making. In a controlled tournament, we test six leading open- and closed-source models, where each agent operates controlling the same character to ensure a fair comparison. The models are prompted to interpret game frames and state data to select their next actions. Unlike static evaluations, LM Fight Arena provides a fully automated, reproducible, and objective assessment of an LMM's strategic reasoning capabilities in a dynamic setting. This work introduces a challenging and engaging benchmark that bridges the gap between AI evaluation and interactive entertainment.
Yushuo Zheng, Tongrui Ye, Zicheng Zhang +3
Shanghai Jiao Tong University, Shanghai, China · Shanghai AI Lab, Shanghai, China · Tongji University, Shanghai, China
Interactive environments offer a practical way to evaluate large language model (LLM) agents, and competitive games extend this evaluation to decisions shaped by competing goals and other agents' choices. Yet existing evaluations confined to individual games or execution frameworks leave unclear which competitive advantages persist across settings. We introduce DECE (Diverse Eurogame Competitive Environments), a unified arena spanning ten Eurogames with shared interfaces and evaluation protocols. These games combine lasting strategic commitments, competition over shared resources, and multiple routes to victory. Our primary study evaluates 12 model-harness configurations, spanning nine models and four execution harnesses, in 480 three-player matches across ten games. The main and supplementary studies comprise 1,032 completed matches in total. These comparisons reveal broad cross-game ranking agreement alongside game-specific strengths. Matched studies show that the same model can perform differently across execution harnesses, adding another source of variation to these profiles. These performance advantages do not consistently translate into better token or cost efficiency, making resource use an additional consideration when comparing configurations. Together, the findings support evaluating complete agent configurations by both their competitive performance across games and the resources needed to achieve it.
Fengrui Zhang, Xiang Li, Ruomeng Ding +6
Alibaba Group · Nanjing University · National University of Singapore +3
Large language models (LLMs) are increasingly deployed as economic agents in marketplaces, auctions, and bidding settings. Anticipating their behavior in any specific deployment is hard. Existing strategic-reasoning benchmarks evaluate models on fixed canonical games. These benchmarks may saturate as the frontier improves, and they do not allow evaluators to generalize with confidence from benchmark performance to the varied and messy strategic environments that actual deployments involve. We introduce GENSTRAT, which uses procedurally generated strategic environments to address these challenges. Concretely, we generate a distribution of two-player zero-sum imperfect-information card games. The generator can draw fresh games on demand, allowing for evergreen evaluation and resistance to contamination. We pair the game distribution with a capability-profile methodology that decomposes model competence across six axes (state space, temporal depth, information sensitivity, opponent modeling, risk, and brittleness). We also introduce a jaggedness measure of within-distribution smoothness that detects when a model's advantage jumps unpredictably between strategically similar games. We sample 50 benchmark games from a 2,000-game generated pool and evaluate nine frontier and open-weight LLMs in a head-to-head tournament with over 36,000 matches. Newer frontier-tier models score higher on average. Beyond that average, models with near-identical overall strength show qualitatively different capability profiles, and two of the top three leaderboard models (gpt-5 and claude) are noticeably more locally volatile than the third (gemini-3.1-pro), despite being close in overall strength. Together, the capability profile and the jaggedness measure give a deployment-relevant diagnostic that the overall ranking alone cannot provide.