Game-theoretic benchmarks claim that large language model (LLM) strategic decision-making varies across game types, interaction structures, and capability dimensions. However, they only evaluate strategic performance across individual fixed models. In parallel, contemporary LLM routers pick models under quality-cost trade-offs, but with no notion of a strategic structure. They do not account for an opponent that might be unknown, and that can adapt over time and shift the feedback signal. To address this, we extend the game-theoretic evaluations and define strategic selection as the meta game layered on top of the base game. In each round, a strategic selector fields an LLM from a portfolio of models to navigate a game payoff matrix that is latent and learned online. We find that the evaluation signal is game-specific, and that extracting a reliable signal requires a much higher number of seed runs N compared to traditional benchmarking setups. Evaluating across 5 games and 10 LLMs, we show a model's strategic competence varies significantly depending on the game setup and the opponent faced. Building on this, we implement a suite of selector methods and find that approaches that rely on query embeddings or LLM self-reflection do not consistently capture the underlying strategic signal. A lightweight multi-armed bandit algorithm performs more reliably across games, though it still falls short of the best fixed model. Finally, we show that simple non-stationary and adversarial bandit variants fail to outperform a stationary bandit baseline, pointing to a gap in how current methods handle a shifting opponent in strategic settings.
Figures & tables
Figure 1: Left: A strategic selector i plays a repeated game over T rounds against an unknown opponent −i and observes the game outcome ξi,t . The \textscUpdate(θi,t,xi,t,ξi,t) function is applied at every round t . Right: Strategic performance (game utility v ) varies across games and models.
Method
Cost-Aware
Opp.-ID
Online
Embed.
Non-Stat.
Source
Fixed Model
×
×
×
×
×
—
Per-Opponent Full-Information
✓
✓
×
×
×
—
Implemented selectors
LLM Selector
✓
×
✓
×
×
Shinn et al. [21]
RouterDC
✓
×
×
✓
×
Chen et al. [5]
Causal LLM Router
✓
×
✓
✓
×
Tsiourvas et al. [26]
Table 1: Implemented suite of selector methods. Cost-Aware : the method accounts for a quality-cost trade-off when selecting a model. Opp.-ID : the opponent’s identity is revealed to the selector. Online : the method learns online (round-by-round) from game outcomes. Embed. : the method encodes context into embeddings before feeding them into the predictor. Non-Stat. : the method is designed to handle a non-stationary environment.
Figure 2: (a) Model scale does not reliably translate into strategic capability. (b) Models differ in strategic disposition. Each point shows a model at the projection of its seven in-game disposition scores onto the first two principal components. Granite , Mistral , Llama , Qwen3.5 .
Model
Chicken Game
Colonel Blotto
Cournot Duopoly
Stag Hunt
Vickrey Auction
Granite4.1 3B
87.5 ± 0.3
50.6 ± 0.3
26.0 ± 1.2
74.3 ± 2.9
74.3 ± 0.6
Granite4.1 8B
69.6 ± 3.0
50.5 ± 0.3
7.8 ± 1.5
65.6 ± 2.2
73.7 ± 0.6
Granite4.1 30B
87.3 ± 0.3
50.4 ± 0.2
13.2 ± 1.5
72.9 ± 2.9
69.4 ± 0.5
Ministral3 3B
86.4 ± 0.7
48.5 ± 0.5
18.4 ± 2.0
73.0 ± 2.8
72.0 ± 0.6
Mistral-Nemo 12B
84.2 ± 1.2
50.3 ± 0.2
20.0 ± 2.3
72.7 ± 2.8
71.9 ± 0.6
Mistral-Small 3.2 24B
82.2 ± 1.7
49.7 ± 0.3
35.2 ± 2.8
56.1 ± 1.2
72.5 ± 0.6
Table 2: Normalised in-game utility v↑ acquired by each model across 5 games. We report the mean ± 95% CI, ×100 , aggregated across all pairwise matchups, using T=1 round, N=300 seeds per run, and highlight the best score for each game.
Selector
Chicken Game
Colonel Blotto
Cournot Duopoly
Stag Hunt
Vickrey Auction
Reports normalised gain g↑
Per-Opponent Full-Information
92.8 ± 0.4
73.0 ± 0.2
80.7 ± 1.5
92.2 ± 1.0
86.7 ± 0.5
Thompson Sampler (TS)
85.1 ± 0.5
71.2 ± 0.2
47.7 ± 0.8
76.1 ± 0.8
74.8 ± 0.1
RouterDC
84.3 ± 0.1
68.6 ± 0.1
36.2 ± 0.1
74.4 ± 0.8
72.8 ± 0.2
Causal LLM Router
78.6 ± 0.7
63.2 ± 0.5
31.3 ± 0.7
74.0 ± 0.9
64.8 ± 0.5
LLM Selector
61.0 ± 0.6
46.9 ± 0.3
26.9 ± 0.5
46.6 ± 1.2
54.4 ± 0.2
Table 3: Normalised net gain g↑ (top) and in-game utility v↑ (bottom) acquired by each selector, against fixed opponents and across 5 games. We show the mean ± 95% CI, ×100 , averaged across 10 runs (the size of our portfolio), using T=300 rounds, N=1 seed per run, and highlight the best score for each game. Selectors’ learning signal was set as their per-round gain gi,t .
Selector
net gain g↑
in-game utility v↑
usage cost c ( ×10−3)\downarrow$
Thompson Sampler (TS)
72.4 ± 0.8
47.1 ± 1.7
0.027 ± 0.001
Sliding-Window TS
65.9 ± 0.7
33.2 ± 1.4
0.017 ± 0.001
Discounted TS
63.1 ± 0.8
27.5 ± 1.6
0.017 ± 0.001
EXP3
60.2 ± 0.7
22.1 ± 1.4
0.020 ± 0.001
RouterDC
59.6 ± 0.3
19.3 ± 0.5
0.002 ± 0.000
Causal LLM Router
59.1 ± 0.7
19.7 ± 1.3
0.019 ± 0.001
Table 4: Normalised net gain g↑ , in-game utility v↑ , and raw usage cost c↓ acquired by each selector, against other selectors and in Cournot Duopoly. We report the mean ± 95% CI, ×100 , aggregated across all pairwise matchups, using T=300 rounds, N=1 seed per run, and highlight the best score for each metric. Selectors’ learning signal was set as their per-round gain gi,t .
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
Meta Game Γ (LLM selection game)
i,−i
player indices, where −i denotes the opponent of i
t=1,…,T
round index of the game
N
number of seeds in the game (independent replications)
Si={1,…,Mi}
finite set of pure strategies, i.e. player i ’s model portfolio
σi∈Δ(Si)
mixed strategy for player i , a probability simplex over Si
Appendix
Table A.1 : Glossary of notation.
Figure B.1 : Statistical analysis of signal and noise across 5 games and 4 model families. (a) Signal, noise, and SNR per game, n=300 . (b) Similar to Heineman et al. [8] (R=0.79) we show that our correlation holds at r≈0.81 from n=75 onwards.
Figure B.2 : Statistical analysis of signal and noise across 5 games and 4 model families. (a) Fraction of all 45 reported pairwise gaps that are statistically detectable. (b) Decision accuracy, mean over 30 subsample draws per n .
Model
Hugging Face repository
granite4.1:3b
ibm-granite/granite-4.1-3b
granite4.1:8b
ibm-granite/granite-4.1-8b
granite4.1:30b
ibm-granite/granite-4.1-30b
llama3.1:8b-instruct-q4_K_M
meta-llama/Llama-3.1-8B-Instruct
llama3.3:70b
meta-llama/Llama-3.3-70B-Instruct
ministral-3:3b
mistralai/Ministral-3-3B-Instruct-2512
Appendix
Table C.1 : Model portfolio used across experiments.
Game
Structure
Source
Chicken Game
anti-coordination, brinkmanship
https://en.wikipedia.org/wiki/Chicken_(game)
Colonel Blotto
zero-sum resource allocation
https://en.wikipedia.org/wiki/Blotto_game
Cournot Duopoly
simultaneous quantity competition
https://en.wikipedia.org/wiki/Cournot_competition
Stag Hunt
coordination, risk dominance
https://en.wikipedia.org/wiki/Stag_hunt
Vickrey Auction
second-price sealed-bid auction
https://en.wikipedia.org/wiki/Vickrey_auction
Appendix
Table C.2 : The five base games G used across experiments, all played between 2 players.
Game
Disposition
Definition
PC1
PC2
Chicken Game
Risk appetite
Fraction of rounds choosing dare over swerve.
+0.35
−0.12
Cournot Duopoly
Cooperativeness
Collusion index; 1 = joint-profit quantity, 0 = at or beyond Nash.
−0.45
−0.41
Aggressiveness
Own minus opponent quantity, as a share of the maximum quantity.
+0.49
+0.35
Stag Hunt
Risk appetite
Fraction of rounds choosing stag over the safe hare.
−0.38
+0.50
Cooperativeness
Fraction of rounds landing on mutual stag.
−0.34
+0.54
Vickrey Auction
Risk appetite
Mean own bid as a share of the maximum bid.
+0.38
+0.29
Appendix
Table D.1 : Strategic disposition metrics and their loadings on the first two principal components. All metrics are averaged across rounds and lie in [0,1] .
Figure D.1 : Normalised in-game utility v↑ against the disposition. Dashed lines show the linear fit and r its Pearson correlation. Granite , Mistral , Llama , Qwen3.5 .
Figure D.2 : Strategic dispositions inferred from in-game behaviour for each model family. Each axis is min-max normalised across the 10 models of the portfolio. Granite , Mistral , Llama , Qwen3.5 .
Model
Chicken Game
Colonel Blotto
Cournot Duopoly
Stag Hunt
Vickrey Auction
Reports normalised gain g↑
Granite4.1 3B
93.8 ± 0.1
75.2 ± 0.1
63.0 ± 0.6
87.2 ± 1.4
87.2 ± 0.3
Granite4.1 8B
82.3 ± 1.5
74.5 ± 0.1
46.9 ± 0.8
72.0 ± 1.2
85.8 ± 0.3
Granite4.1 30B
87.0 ± 0.1
74.0 ± 0.1
48.6 ± 0.9
78.1 ± 1.4
80.1 ± 0.3
Ministral3 3B
86.5 ± 0.4
73.4 ± 0.2
53.5 ± 1.0
78.2 ± 1.4
84.2 ± 0.3
Mistral-Nemo 12B
92.1 ± 0.6
75.0 ± 0.1
59.4 ± 1.2
86.2 ± 1.4
85.8 ± 0.3
Appendix
Table D.2 : Net gain, normalised utility, and normalised cost acquired by each model across 5 games. We report the mean ± 95% CI, ×100 , aggregated across all pairwise matchups, using T=1 round, N=300 seeds per run. Highlight marks the best score per game within each metric block.
Figure D.3 : Model elo score (bootstrapped) vs. scale. Granite , Mistral , Llama , Qwen3.5 .
Method
Chicken Game
Colonel Blotto
Cournot Duopoly
Stag Hunt
Vickrey Auction
Reports normalised gain g↑
Per-Opponent Full-Information
92.8 ± 0.4
73.0 ± 0.2
80.7 ± 1.5
92.2 ± 1.0
86.7 ± 0.5
Granite4.1 3B
89.7 ± 0.1
71.6 ± 0.1
64.0 ± 0.5
89.2 ± 0.8
83.4 ± 0.2
Granite4.1 8B
84.5 ± 0.8
70.5 ± 0.1
49.8 ± 0.5
81.1 ± 0.7
82.0 ± 0.2
Granite4.1 30B
77.3 ± 0.1
59.1 ± 0.1
40.9 ± 0.5
76.6 ± 0.8
68.6 ± 0.2
Ministral3 3B
86.5 ± 0.2
67.3 ± 0.2
53.3 ± 0.7
83.6 ± 0.8
78.9 ± 0.2
Appendix
Table D.3 : Normalised net gain g↑ (top) and in-game utility v↑ (bottom) acquired by each selector and fixed model from full pairwise matchups against each other, and across 5 games. We show the mean ± 95% CI, ×100 , using T=300 rounds, N=1 seed per run, and highlight the best score for each game. Selectors’ learning signal was set as their per-round gain gi,t .
Large language models (LLMs) are increasingly deployed as economic agents in marketplaces, auctions, and bidding settings. Anticipating their behavior in any specific deployment is hard. Existing strategic-reasoning benchmarks evaluate models on fixed canonical games. These benchmarks may saturate as the frontier improves, and they do not allow evaluators to generalize with confidence from benchmark performance to the varied and messy strategic environments that actual deployments involve. We introduce GENSTRAT, which uses procedurally generated strategic environments to address these challenges. Concretely, we generate a distribution of two-player zero-sum imperfect-information card games. The generator can draw fresh games on demand, allowing for evergreen evaluation and resistance to contamination. We pair the game distribution with a capability-profile methodology that decomposes model competence across six axes (state space, temporal depth, information sensitivity, opponent modeling, risk, and brittleness). We also introduce a jaggedness measure of within-distribution smoothness that detects when a model's advantage jumps unpredictably between strategically similar games. We sample 50 benchmark games from a 2,000-game generated pool and evaluate nine frontier and open-weight LLMs in a head-to-head tournament with over 36,000 matches. Newer frontier-tier models score higher on average. Beyond that average, models with near-identical overall strength show qualitatively different capability profiles, and two of the top three leaderboard models (gpt-5 and claude) are noticeably more locally volatile than the third (gemini-3.1-pro), despite being close in overall strength. Together, the capability profile and the jaggedness measure give a deployment-relevant diagnostic that the overall ranking alone cannot provide.
Large Language Models (LLMs) have achieved strong performance on static reasoning benchmarks, yet their effectiveness as interactive agents operating in adversarial, time-sensitive environments remains poorly understood. Existing evaluations largely treat reasoning as a single-shot capability, overlooking the challenges of opponent-aware decision-making, temporal constraints, and execution under pressure. This paper introduces Strategic Tactical Agent Reasoning (STAR) Benchmark, a multi-agent evaluation framework that assesses LLMs through 1v1 zero-sum competitive interactions, framing reasoning as an iterative, adaptive decision-making process. STAR supports both turn-based and real-time settings, enabling controlled analysis of long-horizon strategic planning and fast-paced tactical execution within a unified environment. Built on a modular architecture with a standardized API and fully implemented execution engine, STAR facilitates reproducible evaluation and flexible task customization. To move beyond binary win-loss outcomes, we introduce a Strategic Evaluation Suite that assesses not only competitive success but also the quality of strategic behavior, such as execution efficiency and outcome stability. Extensive pairwise evaluations reveal a pronounced strategy-execution gap: while reasoning-intensive models dominate turn-based settings, their inference latency often leads to inferior performance in real-time scenarios, where faster instruction-tuned models prevail. These results show that strategic intelligence in interactive environments depends not only on reasoning depth, but also on the ability to translate plans into timely actions, positioning STAR as a principled benchmark for studying this trade-off in competitive, dynamic settings.
Yang Li, Xing Chen, Yutao Liu +5
Tsinghua University · Ocean University of China (OUC) · CAICT +4
Large language models (LLMs) are increasingly tasked with strategic decision-making under incomplete information, such as in negotiation and policymaking. While LLMs can excel at many such tasks, they also fail in ways that are poorly understood. We shed light on these failures by uncovering two fundamental gaps in the internal mechanisms underlying the decision-making of LLMs in incomplete-information games, supported by experiments with open-weight models Llama 3.1, Qwen3, and gpt-oss. First, an observation-belief gap: LLMs encode internal beliefs about latent game states that are substantially more accurate than their own verbal reports, yet these beliefs are brittle. In particular, the belief accuracy degrades with multi-hop reasoning, exhibits primacy and recency biases, and drifts away from Bayesian coherence over extended interactions. Second, a belief-action gap: The implicit conversion of internal beliefs into actions is weaker than that of the beliefs externalized in the prompt, yet neither belief-conditioning consistently achieves higher game payoffs. These results show how analyzing LLMs' internal processes can expose systematic vulnerabilities that warrant caution before deploying LLMs in strategic domains without robust guardrails.