Training language-model agents to play long-horizon strategy games is hard because deterministic feedback is sparse and often arrives only at the end of a game. In-game scores can offer dense signals, but they do not reliably predict future outcomes: in 8-68% of position pairs across eight strategy games, the higher-scoring position leads to the worse outcome. We term this phenomenon scoreboard traps. In Freeciv, an open-source turn-based empire-building game where several players manage units, cities, and diplomacy over hundreds of turns, we look into better reward signals for gameplay agents. To facilitate this, we build CivHarness, an open system that saves current game states, branches on different decisions, and replays the game to future turns to provide measurable long-horizon feedback from saved positions. We use CivHarness to train CivTelescope, an LLM Judge that predicts which one of two game positions will yield the better outcome. On scoreboard-trap pairs, CivTelescope is 1.3-2.0 times more accurate than off-the-shelf language models, and it generalizes to new game starts, later turns, and unseen rules. We then train a Freeciv gameplay agent with CivTelescope as a dense reward, and call this recipe CivMarshmallow. When used in the middle stage of the game where scoreboard traps are common, CivMarshmallow reaches an average training phase gain of 8.94 points, 77% more than the best baseline, the end-of-game reward. Over the full 120-turn game, CivMarshmallow uses a hybrid reward that starts from the scoreboard or the end-of-game reward and shifts to CivTelescope as the game advances; this hybrid reward scores highest among the rewards we compare.
Figures & tables
Figure 1: Freeciv and the scoreboard trap. (a) Current scoreboard score (solid) and future score at turn 120 (dashed) in Freeciv games A ( gray ) and B ( red ). A leads the scoreboard until turn 50, while B leads the future score during the trap ( yellow ). (b) Frontier LLMs fail at identifying scoreboard traps given a pair of two positions.
Figure 2: Scoreboard traps. (a) Trap rate across eight strategy games. (b) Trap rate by game progress; darker red is a higher rate, scaled within each game.
Figure 3: Overview of CivHarness , CivTelescope , and CivMarshmallow . CivHarness branches saved Freeciv states and exchanges player-visible observations and actions with the agent. CivTelescope , a pairwise judge trained on replay-labeled trap pairs, gives a potential Φ(s) through a reference set. CivMarshmallow trains the policy in four phases with CivTelescope reward, and with a hybrid reward for full games.
Method
Held-out games ↑
Unseen ruleset ↑
Baselines
scoreboard
0
0
7B backbone , zero-shot
0.260
0.241
GPT-5.6-luna , zero-shot
0.258
0.203
GPT-5.5 , zero-shot
0.262
0.285
GPT-5.6-terra , zero-shot
0.293
0.268
Table 1: Trap accuracy on held-out games (in-domain) and an unseen ruleset; chance is 0.5. ↑ : higher is better. † No significant difference between the two marked values.
Reward
rem20
rem40
rem80
rem120
Baselines
sparse
0.53 (1.1/0.9/ − 0.4)
7.11 (7.2/7.0/7.1)
5.04 (4.7/3.0/7.4)
2.87 (4.3/4.6/ − 0.3)
scoreboard
2.72 (1.4/3.3/3.5)
3.68 (7.0/1.9/2.2)
0.83 ( − 0.1/1.4/1.1)
4.87 (5.8/6.3/2.5)
CivMarshmallow (Ours)
CivTelescope
0.62 (1.9/ − 0.3/0.3)
3.61 (7.4/3.9/ − 0.4)
8.94 (5.4/9.9/11.6)
2.77 (5.1/2.6/0.6)
scoreboard → CivTelescope
–
–
–
5.21 (6.2/5.5/3.9)
Table 2: Held-out phase gain G ( ↑ : higher is better) in each training phase: mean over three training seeds, with per-seed values in parentheses; each value averages the 45 positions of the evaluation set. Bold marks the best mean in each phase.
Method
Terminal score S↑
Eliminated (%) ↓
Invalid actions (%) ↓
Calls per decision
Prompting agents
Base model (Qwen3-8B)
7.38
30.0
6.0
1.00
Direct prompting
7.62
26.2
5.7
1.00
BaseLang ( Qi et al., 2024 )
11.65
12.4
16.2
1.00
Mastaba ( Qi et al., 2024 )
11.30
16.2
24.7
4.00
SAGA ( Jin et al., 2026 )
13.23
8.8
41.6
1.10
Table 3: Terminal score S and gameplay statistics on the 45 held-out full-game positions (120 turns, eight games per position); trained policies are evaluated after the final phase ( rem120 ). Eliminated : % of games in which the agent’s player is destroyed; invalid actions : % of decisions rejected by the interface. ↑ / ↓ : higher/lower is better.
Component
Variant
Metric
Result
Label source
Replay outcomes (ours)
Trap accuracy ↑
0.610
GPT-5.6-sol preferences
0.391
Judge input
Full state (ours)
Trap accuracy ↑
0.778
Score numbers masked
0.687
Score numbers only
0.113
Hybrid order
scoreboard → CivTelescope (ours)
Phase gain G at rem120 ↑
5.21
Table 4: Ablations. Each block changes one component and keeps the rest fixed. Trap accuracy is evaluated on the 600 held-out trap pairs. ↑ : higher is better.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Measurement
Positions
Played out by
K
Game length
Decidable pairs
Fig. 2 , civ2civ3 ; Fig. 1 a
40 games, turns 10–115
bots
8
120 turns
cross-game, >2 SE
Fig. 2 , classic
40 games, turns 10–115
bots
8
120 turns
cross-game, >2 SE
Ruleset transfer (Table 1 )
same games, turns 10–65
bots
8
120 turns
cross-game, >2 SE
Table 9
10 games per length
bots
8
60, 70, 80 turns
cross-game, >2 SE
Judge held-out set
bot games; rollouts
bots; the rollout
8; 1
70 turns
cross-game, >2 SE; ≥6
Table 10
training rollouts
the rollout
1
120 turns
same start and turn, ≥6
Appendix
Table 5: The future outcome behind each Freeciv measurement. Every row uses the evaluated player’s terminal score: K branches are played from the position to the configured final turn by the built-in bots at the normal bot difficulty and averaged, or a policy rollout’s own terminal score is taken. The judge’s held-out set is that of Tables 11 and 1 (in-domain column). Cross-game pairs join positions of different games at most ten turns apart; a pair is decidable when its outcome gap exceeds twice the pooled standard error of the branch means, or, for single rollouts, is at least six points.
Chess
Othello
Judge
Non-trap
Trap
Non-trap
Trap
GPT-5.4-mini
0.938
0.157
0.847
0.188
DeepSeek-V3.2
0.862
0.315
0.815
0.263
GPT-5.2
0.892
0.352
0.785
0.637
Claude Sonnet 5
0.883
0.485
0.797
0.522
GPT-5.6-sol
0.677
0.598
0.492
0.813
Appendix
Table 6: Zero-shot accuracy of the five API judges on chess and Othello by pair slice. Each game uses 300 trap and 300 non-trap pairs in both presentation orders, with the Freeciv prompt and only the game name changed; chance is 0.5.
Game (genre)
Visible proxy
Engine
Positions
K
Pairs
Trap rate
Chess (board)
material balance
Stockfish, skill-level noise
541
20
34,024
16.0%
Othello (board)
disc differential
depth-2 search + temperature
420
24
18,334
68.2%
Backgammon (board, dice)
pip-count lead
MCTS-30 + 10% random
187
100
6,587
8.1%
Go (9x9) (board)
area count with komi
MCTS-30 + 10% random
367
100
6,182
38.8%
Hearts (card)
penalty so far vs. table
uniform random
300
100
14,543
17.7%
2048 (single-player puzzle)
score
one-ply greedy + 10% random
381
100
12,042
31.2%
Appendix
Table 7: Trap prevalence in every cross-game bank. Positions are snapshots of self-play games between copies of the named engine (80 chess, 70 Othello, 40 backgammon, 59 Go, 60 Hearts, 60 2048, and 40 Catan games); the oracle replays each position K times to the end with the same engine and takes the mean outcome. Pairs are built as in Freeciv: across games, bounded move gap, decidable when the oracle gap exceeds twice its pooled standard error; the trap rate is taken over decidable pairs whose visible proxy is not tied. Zero-shot judges are evaluated on the chess and Othello banks only (Table 6 ).
Progress (%)
0–10
10–20
20–30
30–40
40–50
50–60
60–70
70–80
80–90
90–100
Backgammon
10
5
6
8
9
10
9
11
9
6
Catan
25
16
15
16
12
11
11
12
13
12
Chess
6
19
21
17
12
9
5
3
4
2
Hearts
–
18
20
23
22
18
12
8
7
4
Freeciv (civ2civ3)
55
49
40
35
29
28
24
18
10
8
Freeciv (classic)
41
46
46
35
31
28
24
17
11
7
Appendix
Table 8: Trap rate (%) in each progress bin of Figure 2 b. Progress is the mean position of a pair divided by the typical length of a complete game; a dash marks a bin with fewer than 30 pairs: no points are scored while cards are passed in Hearts, and the Freeciv banks cover turns 10–115 of 120-turn games (Table 5 ).
Game length
Scoreboard corr.
Oracle corr.
Turns won by oracle
Trap rate
60 turns
0.81
0.98
7 / 10
30%
70 turns
0.53
0.95
11 / 12
31%
80 turns
0.12
0.58
12 / 14
54%
Appendix
Table 9: The scoreboard predicts the terminal score less well in longer games. For ten games at each game length, we correlate the current scoreboard and the replay-oracle value ( K=8 branches) with the game’s terminal score at every fifth turn, and average the correlations over turns. “Turns won” counts the sampled turns at which the oracle value correlates at least as well as the scoreboard; the trap rate is the share of decidable position pairs whose scoreboard order disagrees with their terminal-score order.
Turn bucket
Same-start trap rate
95% interval
Same-start pairs
Cross-start trap rate (pairs)
10–29
0.233
[0.173, 0.304]
7,255
0.298 (95,108)
30–49
0.179
[0.152, 0.208]
31,384
0.113 (216,768)
50–69
0.219
[0.199, 0.241]
104,653
0.037 (327,427)
70–89
0.102
[0.091, 0.114]
119,881
0.019 (348,151)
90–119
0.071
[0.063, 0.079]
389,279
0.010 (742,316)
Appendix
Table 10: Within-game trap rate on the training rollouts. Pairs of rollouts from the same start, compared at the same turn, whose realized terminal scores differ by at least six points; a trap is a pair whose leader on the visible score finishes behind, with tied visible scores dropped. Cross-start pairs are drawn from the same rollouts at the same turn under the same rule. Intervals are cluster bootstraps over 640 arm-by-start clusters.
Judge
Non-trap pairs
Trap pairs
Warned
Times more accurate
Qwen2.5-7B-Instruct (zero-shot backbone)
0.766
0.260
—
1.98 [1.22, 3.27]
GPT-5.6-luna
0.792
0.258
0.317
2.00 [1.27, 3.25]
GPT-5.5
0.805
0.262
0.347
1.96 [1.25, 3.10]
GPT-5.6-terra
0.823
0.293
0.355
1.64 [1.03, 2.64]
GPT-5.6-sol
0.802
0.330
0.378
1.35 [0.87, 2.13]
GPT-6-astra
0.818
0.343
0.378
1.26 [0.76, 2.03]
Appendix
Table 11: Trap accuracy on the judge’s held-out set: 300 trap and 300 non-trap pairs from games and rollouts excluded from training, both presentation orders, content-level. Chance is 0.5. Warned repeats the trap pairs with a sentence in the prompt saying that the visible score can mislead and that the future outcome decides. The last column is how many times more accurate the judge is than the model on the same trap pairs, with a 95% paired bootstrap interval clustered over source games.
Replay-grounded judge
Scoreboard
Turn bucket
Median error
Spearman ρ
Median error
Spearman ρ
10–29
7.8 / 9.6
0.09 / 0.10
15.0 / 16.0
0.34 / 0.36
30–49
4.8 / 6.2
0.42 / 0.41
12.0 / 12.0
0.64 / 0.65
50–69
3.9 / 5.1
0.60 / 0.54
8.0 / 8.0
0.78 / 0.78
70–89
3.6 / 3.9
0.64 / 0.65
6.0 / 6.0
0.86 / 0.87
90–109
2.9 / 3.0
0.74 / 0.76
3.0 / 3.0
0.92 / 0.92
Appendix
Table 12: Offline window diagnostic on rem120 rollouts. Per turn bucket, the median absolute error and the Spearman correlation of each potential against the realized terminal score, for two training seeds (a / b). The judge’s error is below the scoreboard’s from the first bucket, but its rank correlation only exceeds 0.4 from the 30–49 bucket onward, and the scoreboard becomes an identity in the last bucket.
Long-horizon strategic planning in complex strategy games requires coordinating tightly coupled decision domains, including technology, economy, diplomacy, and military, across hundreds of turns under imperfect information. Existing LLM-based agents face three challenges in this setting. They recover little relational structure, such as distance or threat, from raw coordinate observations. They serialize the entire growing game state into every prompt and plan all domains in a single output, so the context eventually overflows and urgent domains crowd out long-term ones. The only reward is a lagged final score, which provides no progress signal within or across games.<br/><br/>We present SAGA, an LLM multi-agent framework that addresses these challenges with three mechanisms: a Map-Semantic Scene Graph that renders typed entity relations as concise per-entity text, a Tool-Augmented Planner that retrieves state on demand and routes disjoint per-domain directives to specialist controllers, and a Dual-Horizon Feedback Loop that sets intermediate goals within a game and distills causal lessons across games. On CivRealm, a benchmark built on the strategy game FreeCiv, SAGA attains the highest mean final score among six methods, the statistically strongest gains on infrastructure, and a 27% reduction in output tokens. With cross-game evolution enabled, it reaches the highest score over five successive games.
Tianyu Jin, Shuo Chen, Yida Wang +6
Beijing University of Posts and Telecommunications · Beijing Institute for General Artificial Intelligence (BIGAI)
Training large language models (LLMs) to act in long-horizon games is a promising step toward generalist decision-making, yet reinforcement learning with verifiable rewards (RLVR) relies on sparse final rewards that reveal little about which decisions determine success. Denser process signals could supply this missing turn-level credit, but existing sources are hard to keep both cheap and accurate. We observe that changes in a game solver's state value reveal whether an action advances the state toward success. Building on this insight, we propose CAST (Credit Assignment from Solver Teachers), which converts these value changes into solver advantages and injects them into RLVR as turn-level signals. We further show that, under a soft-optimal solver assumption, maximizing the solver advantage is equivalent to on-policy distillation from the solver, requiring only scalar values rather than teacher logits. Across Sokoban, Minesweeper, and Rush Hour, CAST outperforms all trained baselines on every game under both in-domain and unseen-difficulty evaluation and achieves the highest average zero-shot performance on ALFWorld and WebShop. Our code is available at https://github.com/Wloner0809/CAST.
Yu Wang, Yi-Kai Zhang, Wentao Shi +8
University of Science and Technology of China · 4Meituan, China · 2Nanjing University +1
Evaluating strategic decision-making in LLM-based agents requires generative, competitive, and longitudinal environments, yet few benchmarks provide all three, and fewer still offer evaluation signals rich enough for long-horizon, multi-agent play. We introduce CivBench, a benchmark for LLM strategists (a model under an agentic setup) in multiplayer Civilization V. Because terminal win/loss is too sparse a signal in games spanning hundreds of turns and multiple opponents, CivBench trains estimators of a state-value function, the probability that a player eventually wins given the game state. cross 307 games with 7 LLMs and multiple CivBench agent conditions, we demonstrate CivBench's potential to estimate strategic capabilities in Civilization V as an unsaturated benchmark, reveal model-specific effects of agentic setup, and outline distinct strategic profiles not visible through outcome-only evaluation.