SpireBench: Evaluating Long-Horizon Strategic Play of Agentic Models with Slay the Spire

Scott C. Lowe; Sander W. Keemink

Figure 5: Our harness. Game states become text, each action’s effect attributed to its tool call; a router sends each part of the game to the model or a symbolic player; in-game names are (optionally) reskinned; tool calls are mapped back and validated. The prompt runs from cached layers to turn messages. Bottom: when the model plans, forecasts and writes notes.

Figure 6: The harness as a sequence of messages, with time running downwards. In combat the model returns a whole turn as several tool calls in one reply; the harness sends them to the game one command at a time, diffs the state after each, and returns each call’s own effect as that call’s result, followed by each enemy’s action and the next turn’s state. At the end of each floor the model forecasts its chances and may update its run notes; at the end of the run it consolidates them into the memory bank that the next run’s system prompt carries. Code on the right is the shape of what the model sends and receives, with names as it sees them under the reskin.

Kind	None (real)	Themed	Random	Permuted	Permuted and themed
Card	Bash	Trip Attack	Bek	Reckless Charge	Reckless Blow
Card	Strike	Slash	Drakae	†Defend	†Dodge
Mechanic	Block	Guard	Taedoast	Channel	Summon
Status	Vulnerable	Exposed	Baevimi	No Draw	Draw Lock
Relic	Burning Blood	Bloodied Chalice	Vriastild Tousorn	Fossilized Helix	Winged Statuette
Enemy	Gremlin Nob	Goblin Berserker	Slagloath Skial	Centurion	Guardian

Appendix

Table 4: Example entries of the five reskin mappings ( Table 29 ). Under none each entity keeps the name in the second column. Card names are shown verbatim. N.B. There are some in-game effects which act on “cards containing Strike”; a permutation would make that rule name a family the model cannot see, so in both permuted mappings the Strike family is marked with a dagger and the rule refers to it. In the thematic reskin we map all instances of Strike to Slash, avoiding the need for a separate marker.

	Ours	Orak	Bottled AI
Characters
Ironclad, Silent, Defect	✓	✓	✓
Watcher	✓	✗	✓
Combat: what is seen
Hand, energy, HP, Block, relics	✓	✓	✓
Enemy intents and powers	✓	✓	✓

Appendix

Table 5: Which parts of the game each harness covers. ✓ : covered (for our harness and Orak, the LLM sees/decides it) rule bot : Orak’s rule-based bot decides it, without the LLM. ✗ : not covered. For Bottled AI, a symbolic player with no model, ✓ means it handles it, and a percentage is the share it handles.

	Ironclad		Silent		Defect		Watcher		Colourless		All cards
	(75)		(75)		(75)		(75)		(51)		(351)
Ours	75	(100%)	75	(100%)	75	(100%)	75	(100%)	51	(100%)	351	(100%)
Bottled AI	63	(84%)	66	(88%)	68	(91%)	70	(93%)	33	(65%)	300	(85%)
Orak	68	(91%)	66	(88%)	72	(96%)	67	(89%)	45	(88%)	318	(91%)

Appendix

Table 6: Cards each harness supports in full, by card pool. A card counts if a player of that character could use it in full; which characters each harness can play, and which parts of the game, are in Table 5 .

Model	Developer	Reached through	Model ID	Reasoning
Qwen3.8 27B	Alibaba	self-hosted	Qwen3_8-27B	xhigh
Qwen3.6 35B-A3B	Alibaba	self-hosted	Qwen3_6-35B-A3B	xhigh
Qwen3.5 122B-A10B	Alibaba	self-hosted	Qwen3_5-122B-A10B	xhigh
Claude Fable 5.1	Anthropic	Anthropic API	claude-fable-5-1	max
Claude Opus 5	Anthropic	Anthropic API	claude-opus-5	max
Claude Sonnet 5	Anthropic	Anthropic API	claude-sonnet-5	max

Appendix

Table 7: The models of Table 1 : developer, how each was reached, the model ID sent, and the reasoning setting it ran at in the main table (an effort level, or a thinking budget for Claude Haiku 4.5). Self-hosted models were served through an OpenAI-compatible endpoint under the deployment name given. Every main-table run was played between 19 and 25 September 2026. N/A : No reasoning control available; the model runs at its default.

Figure 7: Human performance at A20 with the Ironclad against hours of the game played, from the telemetry in Table 10 : mean acts cleared per run (left) and the share of runs that kill the Heart (right). Each point is a bin of Table 10 at its median hours; bins with fewer than 2,000 runs are labelled with their count. Play time resets with a new save profile, so the lowest bin likely holds experienced players on fresh profiles.

	Agents			Humans
	All	Died	Passed	All	Died	Passed
HP (% of max)	82	75	86	62	56	65
Campfires spent resting (%)	68	75	63	23	26	22
excluding Recall (%)	72	81	67	23	27	22
Elites fought	1.25	1.21	1.28	1.96	1.85	2.01
Fights with no card taken (%)	8	12	6	23	26	21

Appendix

Table 11: State and Act 1 choices on entering the Act 1 boss (floor 16), for the runs that reached it, split by whether the run then died at the boss. Campfire, fight and potion rates are pooled over campfires, fights and potions on floors 1–15 (potions 0–15); HP, elites, deck size and potions used at the boss are means over runs.

	Share of deaths (%)		Death rate on reaching (%)
Floor	Agents	Humans	Agents	Humans
7	11	4	12	4
16 (Act 1 boss)	27	23	41	32
22	7	3	20	6
33 (Act 2 boss)	6	8	37	34

Appendix

Table 12: Where runs end: the share of all deaths that fall on a floor, and the death rate among the runs that reached it. Agents: 196 deaths in 200 runs (the rest cleared Act 3, one of them going on to kill the Heart). Humans: 195,533 deaths in 206,082 runs.

Figure 8: What runs did on each act's 15 map floors, over the runs that reached that act's boss (floors 16, 33 and 50): the telemetry population (143,146, 47,619 and 22,729 runs), the four Twitch streamers (788, 671 and 603) and the LLM agents of Table 1 in three groups by mean acts cleared: the top 4 (36, 23 and 10 runs), the next 8 (65 and 7) and the bottom 8 (26). A group is drawn for an act only when at least five of its runs reached that boss. Shops and unknown rooms count the map nodes entered; elites fought and hallway fights the rooms entered. Rest-site and card-reward shares pool the visits and fights of all runs in the group.

Figure 9: Relics, cards in the deck and upgraded cards at the end of each floor survived: median (lines) and interquartile range (bands), for the telemetry population (dashed), the four Twitch streamers, the top 4 LLM agents of Table 1 by acts cleared and the other 16. A floor is drawn only while at least 20 human or 8 LLM runs are alive on it. Human counts are reconstructed backward from each run's final deck and relics, removing every change the record itemizes after the floor.

Figure 10: Act 1 choices and combat against mean acts cleared, one point per LLM agent of Table 1 with at least three runs reaching the Act 1 boss (16 models), over those runs; lines are the telemetry population (dashed) and the four Twitch streamers.

\rho

is Spearman's correlation across the models and

p

a two-sided permutation test. HP change per hallway fight is the net HP change over a hallway-fight floor, including the relic heal after it, computed the same way for humans. Last panel: each model's rate of beating the Act 1 boss against the rate expected from its runs' states on reaching it, by a logistic model fitted on the 143,071 telemetry runs that reached it with a complete state record; the dotted line is equality.

Figure 11: Do models learn from their memories? Mean acts cleared per run over the first five runs of each chain (open circle) against the last five (arrowhead), by memory mechanism; a circle alone means no change. The two halves are different seeds, so the no-memory cohort on the same seeds (grey) is the control for how hard each half is: learning shows as an arrow further right than the grey one. Only cohorts with all 10 seeds are shown. For reference, the top row is one of the authors (a non-professional player) over their first ten A20 Ironclad runs, playing the unmodified game without a harness or memory bank; with five runs per half, a single win moves a half’s mean by 0.8 acts.

	Top 4	Next 8	Bottom 7	$\rho$	$p$
Damage taken per hallway fight	8.2	9.6	14.0	$-0.87$	$<0.001$
Turns per hallway fight	3.09	3.58	4.06	$-0.68$	0.002
Damage taken per hallway turn	2.65	2.77	3.43	$-0.57$	0.012
Damage dealt per energy on attacks	7.5	6.7	5.8	$+0.91$	$<0.001$
Damage dealt per energy, all cards	5.7	4.7	3.8	$+0.76$	$<0.001$
Damage dealt / attack damage in hand	0.95	0.86	0.74	$+0.65$	0.003

Appendix

Table 14: Combat play in Act 1 by tier of Table 1 (top four, next eight, bottom seven with full logs), with Spearman's

\rho

against mean acts cleared over the 19 models and its permutation

p

. Hallway fights are ordinary combats. Block into no attack: turns on which a block card is played although no enemy intends to attack. Over-block: a block-only card played beyond the incoming damage while an attack was playable. Missed lethal: turns with one enemy left and enough damage in hand to kill it. The lower block shows measures that do not separate the tiers (

p>0.4

).

Figure 12: Effect of difficulty on performance. Final floor reached against the pinned ascension level, for every agent in Table 1 with runs at more than one level, on the held-out seeds under the same configuration as the table. Each level is a separate cohort (its own memory chain). The solid line is the median floor at each level, the blue band the interquartile range, and the dotted lines the lowest and highest floor reached. Background shading marks the floors of each act. Levels with fewer than 10 completed seeds are annotated with their seed count. The last panel is every Ironclad run in Mega Crit’s released telemetry (October–November 2020, final content build; abandoned runs excluded), with the dotted lines at the 5th and 95th percentiles instead, since over tens of thousands of runs per level the extremes are always the first and last floors reachable. Each level there is a different population: a level is unlocked by winning at the one below it, so higher levels hold more experienced players, and only Ascension 20 includes runs that continue to the Heart.

Figure 13: Where runs end. For each agent in Table 1 , the share of its runs that ended in each act, sorted by mean floor (right). Agents with a similar mean floor can differ in whether they reliably reach the next act or split between early deaths and deep runs.

Table 15: How well the models predict their own game. At the end of every floor each model is asked for its chance of surviving the current act and of winning the run; when ending a turn it may also predict its HP at the start of the next turn and at the end of the fight. Same runs as Table 1 . Pred. and Obs. are the mean predicted and observed rates (a floor's act counts as survived if the run went on to clear it; a run is won at the Heart). A prediction is easier the closer it is made to the moment it resolves, and stronger models make more of theirs close to an act's end, so the Brier scores (0 is perfect; always answering 50% scores 0.25), the AUC (0.5 is chance; – where every act had the same outcome) and the fight-end HP error are adjusted for that distance: each is scored over the same mix of distances, pooled over all models ( Fig. 15 ). HP: mean absolute error and signed bias (predicted minus actual; positive is optimistic) in hit points, with a prediction made in the fight that killed the player scored against 0. Next-turn HP is always one turn ahead, and close to arithmetic since enemy intents show the incoming damage. Best and second-best scores among models with all 10 seeds are highlighted.

Figure 14: Calibration against play, for each LLM agent in Table 1 . Calibration error is the reliability term ( Murphy, 1973 ) of the Brier score ( Brier, 1950 ) , in percentage points, over ten bins of stated probability, for the two forecasts made at the end of every floor (surviving the act, winning the run), scored separately and averaged. It is computed per run, not adjusted for how far ahead a prediction was made ( Table 15 ); points are the median over a model's runs and bars the interquartile range. Lower effort levels are drawn smaller and paler, joined to the same model. The Spearman correlation is over the levels in Table 1 , one per model.

Figure 15: Prediction error against distance from the outcome, for the runs of Table 1 . Left: Brier score of the act-survival prediction against the number of floors before the act resolves (at its end if the run cleared it, otherwise where the run died). Middle: the same for the run-win prediction, against floors before the run ends. Right: mean absolute error, in HP, of the predicted HP at the end of the fight against the number of turns still to be played in it. Darker is worse, on a scale per panel; blank cells have fewer than 5 predictions. The bottom row pools all models, and its mix of distances is the one each model is scored over in Table 15 . Fight-end predictions sharpen steeply as the fight ends; act-survival predictions barely sharpen before the act resolves.

Figure 16: As Fig. 14 , for the forecasts made within a floor: the HP a model expects at the start of its next turn and at the end of the fight, both optional when ending a turn. Each is scored by its mean absolute error in HP, divided by that forecast's mean error over the predictions of every model in Table 1 (so 1 is average), and the two are averaged for each run. The division matters: fight-end errors are about ten times next-turn ones, so without it a run that made only next-turn predictions would look excellent. A run with no HP predictions has no score. Within-floor error tracks play more closely than end-of-floor calibration ( Table 16 ).

Forecast	Made at	All predictions	Act 1 only
HP at the start of the next turn	ending a turn	$-0.84$	$-0.86$
HP at the end of the fight	ending a turn	$-0.70$	$-0.82$
Surviving the act	end of a floor	$-0.84$	$-0.77$
Winning the run	end of a floor	$-0.45$	$-0.58$
Both HP forecasts ( Fig. 16 )		$-0.92$	–
Both probabilities ( Fig. 14 )		$-0.72$	–

Appendix

Table 16: Which forecasts track play. The Spearman correlation, over the 20 models of Table 1 , between mean acts cleared and the error of each forecast, pooled over each model's runs: mean absolute error for the two HP forecasts, made when ending a turn, and the calibration error of Fig. 14 for the two probability forecasts, made at the end of every floor. Negative means the models that forecast better play better. Every model plays Act 1, so the Act 1 column compares all of them on forecasts about the same part of the game, while the first column also counts later acts, which only the stronger models reach and where hits are larger. Next-turn HP is close to arithmetic once enemy intents are shown, so its error measures how well a model reads the fight rather than what it knows of its own play. Almost no run is won (1.5% of run-win forecasts), so the run-win error mostly measures how much chance a model gives itself. The bottom rows correlate the scores plotted in the two figures, which combine two forecasts each and are medians over runs.

Table 17: Success at each decision scale of Fig. 1 , for the LLM agents of Table 1 (10 runs each). Turn is the share of combat turns without a rejected action, format error or energy mismatch (a lower bound, since every error is charged to a different turn). Combat is the share of fights won; A1 hallway is the share of ordinary Act 1 fights (not elites or the boss) won, and Boss the share of boss fights won (superscript: boss fights played). Act is the share of acts cleared among acts entered, pooled over Acts 1–3. Run is the share of runs that cleared Act 3.

h_{50}

is the human time at which a logistic in

\log

time, fitted with each scale weighted equally, crosses 50%, with each scale placed at a human time from Mega Crit's telemetry: 10 s for a turn, 10 s per turn of an average fight, the mean of the median times to complete Acts 1–3 (8.8, 11.9 and 20.3 min, taken from players who completed each act), and the median run that kills the Heart (50 min). Human rows come from Appx. D and have no turn or combat data.

Table 20: What the models keep in their memory bank after the tenth seed of Table 1 , classified by hand. Bank is the final number of memories; Created, Rewritten and Deleted count consolidation operations over the ten runs (Rewritten excludes verbatim re-saves). Composition is the share of the final bank in each category (a random 30 per model where the bank is larger): a game fact describes how the game works, a heuristic advises, an absolute rule prescribes without condition, an observation recounts one run; Other is route plans and notes about the harness. Wrong is the share stating a false game fact or a rule that fails in common situations, and H. how many of the coded wrong memories trace to harness text that was itself wrong (a rest-site key label, a state diff across the two final bosses, and debuff expiry shown as an enemy’s action); Absolute the share phrased as ALWAYS, NEVER or MUST anywhere; One run the share drawn from a single run. Models are ordered by mean acts cleared. Operations are read from the full run logs, except for Gemma 4 E4B, whose are read from the harness output log (its full logs are not available).

Table 23: Tool-call errors per 100 model calls, for the LLM agents of Table 1 . No usable call : a response in which no tool call could be executed, because every call was malformed or refused or none could be parsed; the harness asks again. Refused call : one call refused before it reached the game, as illegal on the current screen or with invalid arguments; the model is told why. Energy mismatch : a turn ended with a different amount of energy left than the model declared when ending it, a sign that it lost track of its own turn. Recovered from prose : a call the model wrote as text rather than through the tool interface, which the harness recovered and executed. Errors sums the first three, which are the ones the model is told about. Rates are totals over the cohort divided by its calls. Lowest and second lowest in each column, among agents with all 10 seeds.

Failure	Model (seed, floor)	Excerpt
Deliberating over a routine forecast	Nemotron 3 Nano Omni (S3, 6)	“We need to estimate probabilities. The player is at camp after sleeping, HP 24/75 (32%). Actually after sleeping they healed 22 HP, so HP should be 24+22 = 46? Wait initial HP 24/75, sleeping heals 22 …”
Reasoning and action disagree	Grok 4.6 (S9, 10)	“… I think Blood Temper [Armaments] is the better pick here… I'll go with Blood Temper.” Then skips the reward, writing beside the call: “Skip Pulse and Smash as filler…”
Miscounted turn	GLM-5.3 (S5, 13)	“I miscounted energy! … My error: I planned Trip(2) + Slash(1) + Slash(1) = 4E but only have 3E! Oops.”
Blow-by-blow note	Claude Fable 5.1 (S10, 18)	“CRIMSON WARLOCK [Chosen] solo 100 HP (Act 2 R18 opener), run 10 script: T1 Strong Debuff = Jinx [Hex] … $\to$ T2 Debuff = Fatigue [Weak] 3 on ME + Str 3 to itself …” (280 words)
Learnt, not written down	Nemotron 3 Nano Omni (S3, 2)	“… player is Exposed [Vulnerable], taking 50% more damage, so effective damage = 7 * 1.5 = 10.5, rounded down? Probably 10 or 11.”
Misattributed act heal	GPT-5.6 Terra (S5, end)	“Wanderer's Pack [Tiny House] is an excellent low-HP boss reward: … +5 Max HP/current HP … In this run it converted 16/75 to 65/80 before the next Act-2 fight …”

Appendix

Table 24: Excerpts from the runs of Table 1 , verbatim. Names are as the model saw them under the reskin, with the real game term in brackets. Excerpts are taken from the full run logs and cut at the ellipsis.

Figure 17: Each ablation, model by model. Each grey line is one LLM agent: its mean over the 10 seeds in each condition, shown only where it has all 10 seeds in both that condition and the condition’s baseline, so every comparison is paired. Bars are the mean over those agents, and

n

beneath each bar is how many there are. Baselines are the first bar of each panel, except that the compact summary is read against no memory bank with compact notes. Panels show mean acts cleared per run; the pooled tables are Tables 2 , 27 and 29 . Brackets test each condition against its baseline with a paired sign-flip permutation test over agents (*

p<0.05

; n.s. not significant). Orak is shown for the four models with all 10 of its seeds scored; its full paired comparison is in Table 2 .

Table 25: Effect of the harness, per model (supplementary to Table 2 ); Orak cells score runs killed by Orak's harness as a run lost at the floor it had reached; all harness variants were run without the reskin or a cross-run memory bank. All cells are on A20, with each model at its main-table reasoning setting (grey). Acts: mean number of acts cleared per run. Floor: mean final floor. A1: Act 1 win rate (%). A superscript gives the seed count where fewer than 10 seeds completed. Best and second best are per row, among cells with all 10 seeds.

Table 26: Effect of reasoning effort. Each model at every effort level it was run at, A20 on the held-out seeds with the main-table configuration (themed reskin, xml notes, read-write memory, generic guidance). Columns as in Table 1 , with Acts the mean number of acts cleared per run; Act 4 (a Heart kill) is not shown but counts in Acts. Only levels with all 10 seeds completed are shown, and only models with two or more such levels. Bold marks the best level per model on each column. † Self-hosted; priced at Artificial Analysis headline rates.

Figure 18: Performance against thinking: mean acts cleared per run against thinking tokens per floor (log scale), for the same points as Fig. 2(a) .

Figure 19: How the memory bank grows over a 10-run chain. For each model, top: the number of xml memories held after each run (line), with the memories added (blue, up) and deleted (grey, down) in that run’s consolidation. Bottom: the size of the bank carried into the next run, in thousands of characters, for xml consolidation, the single compact summary, and notes kept verbatim. Size is measured exactly, from the system prompt of the following run, relative to the first run’s empty bank; the bank after run 10 is never read, so that curve ends at run 9.

Table 28: Effect of cross-run memory, per model (supplementary to Table 27 ). Within a run the model writes either keyed xml notes or a single compact summary; between runs they are consolidated into the memory bank (xml memories or one compact summary), kept verbatim (xml notes only), or discarded ( None ). All with the themed reskin and generic guidance. All cells are A20 on the held-out seeds, with each model at its main-table reasoning setting (grey). Acts is the mean number of acts cleared per run, Floor the mean final floor, and A1 the Act 1 win rate (%). A superscript gives the seed count where fewer than 10 seeds completed. Best and second best are per row, among cells with all 10 seeds.

Table 30: Effect of the knowledge reskin, per model (supplementary to Table 29 ). None shows the real game names; Themed is our default Lich's Lair reskin; Random replaces every name with pronounceable nonsense; Permuted swaps each name for another real game name of the same kind; Perm.+themed applies the same permutation in the themed vocabulary. All with xml notes, a read-write memory bank and generic guidance. All cells are A20 on the held-out seeds, with each model at its main-table reasoning setting (grey). Acts is the mean number of acts cleared per run, Floor the mean final floor, and A1 the Act 1 win rate (%). A superscript gives the seed count where fewer than 10 seeds completed. Best and second best are per row, among cells with all 10 seeds.

Table 32: What the system prompt carries, per model, on the 10 evaluation seeds (disjoint from those of Table 1 ). The baseline is our themed reskin; each other arm adds one thing to the reskinned prompt: the enemy compendium ( Bestiary ), the Reddit high-ascension strategy guide, or the Jorbs transcript set (his tutorial series and six hours on the Ironclad, 7 h in all; Sec. C.6 ). Guides and transcripts are renamed by the reskin. All arms take xml notes within a run, write no memory bank, and use generic guidance, with each model at its main-table reasoning setting (grey). Acts is the mean number of acts cleared per run and Floor the mean final floor. A superscript gives the seed count where fewer than 10 seeds completed, and – marks an arm with no valid runs. Best and second best are per row, among cells with all 10 seeds. Pooled over models in Table 31 .

Figure 20: Figure 2(a) with every point named, with its reasoning-effort level and, where it has fewer than 10 seeds, its seed count.

Figure 21: Performance against cost per floor: mean acts cleared per run against mean cost per floor (log scale), for the same points as Fig. 2(a) . † Self-hosted; priced at Artificial Analysis headline rates. ‡ Peak-rate list price, an upper bound.

Table 33: What a run costs, for the agents of Table 1 : wall-clock minutes per seed (one game instance, seeds in series), dollars per run and per 10-seed cohort (priced as in Table 1 ), model calls and tokens per run (input including cached, plus output, in thousands); tool-call errors are in Table 23 . Time depends on the provider's throughput on the day as well as on the model.

SpireBench: Evaluating Long-Horizon Strategic Play of Agentic Models with Slay the Spire

Abstract

Figures & tables

Appendix

Long-Horizon Agent Evaluation

AgenticSTS: A Bounded-Memory Testbed for Long-Horizon LLM Agents

GENSTRAT: Toward a Science of Strategic Reasoning in Large Language Models

Cattle Trade: A Multi-Agent Benchmark for LLM Bluffing, Bidding, and Bargaining

Abstract

Long-Horizon Agent Evaluation

Explore similar work

AgenticSTS: A Bounded-Memory Testbed for Long-Horizon LLM Agents

GENSTRAT: Toward a Science of Strategic Reasoning in Large Language Models

Cattle Trade: A Multi-Agent Benchmark for LLM Bluffing, Bidding, and Bargaining