CivBench: Progress-Based Evaluation for LLMs' Strategic Decision-Making in Civilization V

John Chen; Sihan Cheng; Can Gurkan; Mingyi Lin

cs.AIApr 9, 2026

CivBench: Progress-Based Evaluation for LLMs' Strategic Decision-Making in Civilization V

Authors: John Chen, Sihan Cheng, Can Gurkan, Mingyi Lin

Organizations: Northwestern University

Abstract

Evaluating strategic decision-making in LLM-based agents requires generative, competitive, and longitudinal environments, yet few benchmarks provide all three, and fewer still offer evaluation signals rich enough for long-horizon, multi-agent play. We introduce CivBench, a benchmark for LLM strategists (a model under an agentic setup) in multiplayer Civilization V. Because terminal win/loss is too sparse a signal in games spanning hundreds of turns and multiple opponents, CivBench trains estimators of a state-value function, the probability that a player eventually wins given the game state. cross 307 games with 7 LLMs and multiple CivBench agent conditions, we demonstrate CivBench's potential to estimate strategic capabilities in Civilization V as an unsaturated benchmark, reveal model-specific effects of agentic setup, and outline distinct strategic profiles not visible through outcome-only evaluation.

Figures & tables

Figure 2: Overall (first column) and per-strategy (rest columns) ELO ratings.

Figure 3: Most chosen strategy, most pivoted-to strategy, and best-ELO strategy.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Brier score and log loss by turn-progress decile for all eight models. All models improve as games resolve, with the learned models maintaining the lowest loss throughout.

Figure 5: Late-game log loss by victory type and model.

Figure 6: Within-game rank agreement by turn progress. Agreement is high overall and grows stronger in later phases of the game.

Figure 7: Group permutation importance for AttentionMLP by feature group and eventual victory type. Each cell shows the increase in log loss after permuting one semantic feature group; higher values indicate that AttentionMLP relies more heavily on that group for that victory type.

Figure 9: Strategic commitment relative to Vanilla (positive = more concentrated).

Figure 10: Strategy time allocation (mean share of game time) by player type.

Figure 11: Win probability at strategy pivot.

Figure 12: Pivot-flow heatmaps by player type. Each facet shows from $\rightarrow$ to transition rates per game, with average win probability at pivot annotated in-cell.

Figure 16: Stratified bootstrap (B = 2000) percentile intervals for Bradley-Terry ratings by player type.

Figure 17: Stratified bootstrap pairwise dominance matrix. Each cell shows the fraction of replicates in which the row player’s rating exceeds the column player’s.

Figure 18: Distribution of focus_briefer_count per game for briefed configurations.

Field	Description
FoodStored / FoodPerTurn	Food stored and net food income
ProductionStored / ProductionPerTurn	Production stored and output
GoldPerTurn / SciencePerTurn / CulturePerTurn / FaithPerTurn / TourismPerTurn	Per-turn yields
HappinessDelta	Happiness contribution
RazingTurns / ResistanceTurns	Turns until razing completes / resistance ends
BuildingCount	Number of buildings

Appendix

Table 17

Field	Description
PlayerID	Player this zone belongs to
ZoneID	Zone identifier
Territory	Ownership type of the zone
Dominance	Military control (friendly / contested / enemy)
Domain	Combat domain (land / sea / air)
Posture	Military posture (defensive / offensive / etc.)

Appendix

Table 18

Component	Content
Role	Expert player of Civilization V with Vox Populi mod
Expectation	Delegates all tactical decisions to in-game AI; focuses on macro-level strategy; interacts only through tool calls (no user interaction); may invoke multiple tools per turn
Goals	Call tools to set strategic parameters with written rationale for each decision
Decision constraint	Must call either set-strategy / set-flavors or keep-status-quo each turn

Appendix

Table 19

Tool	Parameters	Description
set-flavors	Grand strategy, 34 flavor weights	Set overall strategic direction and tactical flavor biases
set-persona	26 diplomatic weights (0–10)	Set diplomatic personality parameters
set-relationship	Target player, bias (–100 to 100)	Set relationship modifier toward a specific major civilization
set-research	Technology name	Choose next technology to research
set-policy	Policy name	Choose next social policy to adopt
keep-status-quo	(none)	Maintain current strategic settings unchanged

Appendix

Table 20

Input section	Content
Situation	Turn number, era, civilization traits
Civilization	Owned resources, yields, happiness
Options	Available grand strategies, economic strategies, military strategies, flavor weights
Strategies	Current strategic settings and rationale from the previous turn
Victory Progress	Progress toward each victory condition
Players	Full diplomatic and competitive summary for all known players (opinion scores, relationships, alliances, deals, spy activity, military strength, economic output, era, etc.)

Appendix

Table 21

Aspect	Simple	Briefed
Cities report	Full per-city detail	Routed to briefers; strategist receives summaries
Military report	Full unit/zone detail	Routed to briefers; strategist receives summaries
Events report	Full event log	Routed to briefers; strategist receives summaries
Players report	Full player data	Filtered to strategic-level fields (removes raw opinion scores and spy details)
Briefings section	Not present	Structured briefings from subagent pipeline
Additional tool	–	focus-briefer

Appendix

Table 22

Tool	Parameters	Description
focus-briefer	Mode (Military / Economy / Diplomacy), natural-language instruction	Instruct a specific briefer to focus on particular topics in next turn’s report (stored in working memory)

Appendix

Table 23

Group	Feature (share / absolute)	Description
Science	technologies_gap	Technologies the leader has researched that this player has not
	science_share / science_adj	Science output per turn
Culture	policies_gap	Civics/policies the leader has adopted that this player has not
	culture_share / culture_adj	Culture output per turn
	tourism_share / tourism_adj	Tourism generated; tracks Cultural victory progress
Economy	gold_share / gold_adj	Gold income per turn

Appendix

Table 24

Model	Architecture	Hyperparameters	Training
Baseline	Logistic regression (statsmodels Logit); standardized features; cluster-robust SEs by game_id	23 features (no turn_progress )	Isotonic calibration via 5-fold OOF predictions
XGBoost	XGBClassifier; scale_pos_weight for class imbalance	100 trees, max_depth=3, lr=0.189, subsample=0.910, colsample=0.822, $\lambda$ =16.89	Two-stage: early stopping (10 rounds, 10% holdout), then CalibratedClassifierCV (isotonic, cv=5)
MLP	_UtilityNet : 1 hidden layer (60 units), GELU	loss_tp_alpha =0.414, dropout=0.448	27 epochs, batch=32768, Adam (lr=2.75e-3, wd=2.80e-4)
GroupedMLP	_UtilityNet : 1 hidden layer (222 units), GELU; softmax over (game_id, turn) groups	Group-wise CE; loss_tp_alpha =0.398; dropout=0.211	21 epochs, batch=4096 groups, Adam (lr=7.78e-3, wd=3.11e-3)
InteractionMLP	DeepSets: shared encoder (80 units) $\rightarrow$ mean+max pool $\rightarrow$ concat $\rightarrow$ shared decoder (183 units) $\rightarrow$ group softmax	loss_tp_alpha =0.104, dropout=0.322	29 epochs, Adam (lr=2.60e-4, wd=7.39e-4)
AttentionMLP	Shared encoder (30 units) $\rightarrow$ 1-layer pre-norm MHA (3 heads, attn_dropout=0.259) + residual $\rightarrow$ shared decoder (25 units) $\rightarrow$ group softmax	loss_tp_alpha =0.048, dropout=0.261	13 epochs, Adam (lr=3.03e-3, wd=4.51e-3)

Appendix

Table 25

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,203 papers

Choosing LLMs for Strategic Agent Interactions
Oct 13, 2026Arastun Mammadli, Herbert Woisetschläger, Jawad Fayaz +1Model Selection LLM Decision-Making
University of Exeter United Kingdom · IBM Research Switzerland
GENSTRAT: Toward a Science of Strategic Reasoning in Large Language Models
May 22, 2026Vartan Shadarevian, Kia Ghods, Alex Kenich +1LLM Evaluation Imperfect-Information Games
Princeton University · Google
SAGA: Scene-Aware, Goal-Evolving Agents for Long-Horizon Strategy Game Planning
Jun 29, 2026Tianyu Jin, Shuo Chen, Yida Wang +6Multi-Agent LLM Systems Game-Playing Agents
Beijing University of Posts and Telecommunications · Beijing Institute for General Artificial Intelligence (BIGAI)

Oct 13, 2026J/K move · Enter open · S save

Choosing LLMs for Strategic Agent Interactions

Arastun Mammadli, Herbert Woisetschläger, Jawad Fayaz, Shiqiang Wang

University of Exeter United Kingdom · IBM Research Switzerland

Model Selection LLM Decision-Making AI Agent Benchmarks Game-Playing Agents

Open paper AskPDF arXiv

Game-theoretic benchmarks claim that large language model (LLM) strategic decision-making varies across game types, interaction structures, and capability dimensions. However, they only evaluate strategic performance across individual fixed models. In parallel, contemporary LLM routers pick models under quality-cost trade-offs, but with no notion of a strategic structure. They do not account for an opponent that might be unknown, and that can adapt over time and shift the feedback signal. To address this, we extend the game-theoretic evaluations and define strategic selection as the meta game layered on top of the base game. In each round, a strategic selector fields an LLM from a portfolio of models to navigate a game payoff matrix that is latent and learned online. We find that the evaluation signal is game-specific, and that extracting a reliable signal requires a much higher number of seed runs

N

compared to traditional benchmarking setups. Evaluating across 5 games and 10 LLMs, we show a model's strategic competence varies significantly depending on the game setup and the opponent faced. Building on this, we implement a suite of selector methods and find that approaches that rely on query embeddings or LLM self-reflection do not consistently capture the underlying strategic signal. A lightweight multi-armed bandit algorithm performs more reliably across games, though it still falls short of the best fixed model. Finally, we show that simple non-stationary and adversarial bandit variants fail to outperform a stationary bandit baseline, pointing to a gap in how current methods handle a shifting opponent in strategic settings.

Abstract

LLM Agent Evaluation

Explore similar work

Choosing LLMs for Strategic Agent Interactions

GENSTRAT: Toward a Science of Strategic Reasoning in Large Language Models

SAGA: Scene-Aware, Goal-Evolving Agents for Long-Horizon Strategy Game Planning