Zero-Search Reinforcement Learning for Slay the Spire 2 with Failure-Driven Restarts
Organizations: University of Waterloo
Abstract
Slay the Spire 2 (STS2) is a deck-building roguelike in which a winning run at the highest difficulty takes several hundred decisions, and most losses happen in late boss fights that a learner seldom reaches. We train agents that choose every action with one forward pass of a 6.6M-parameter entity transformer, without lookahead. Training and evaluation run the game's own compiled assembly (v0.107.1) in a headless .NET host. The host replays a run exactly from its seed and actions, so training episodes can restart from states the learner has visited, and a failure-driven curriculum allocates these restarts in proportion to where complete runs are lost. On 12,800 held-out games at Ascension 10 (A10), the released policies win 8.05% averaged over the five characters (Ironclad 15.6%, Regent 12.5%), and AlphaSpire's released networks used without search win 0.55% on a common panel. Longer A10 training raised Ironclad from 12.4% to 15.6% and Defect from 3.0% to 8.1% on these games. At Ascension 0 the released policies win 80.2%, level with the 80.3% that the tree-search agent Spire Mentor reports, while Mentor leads by 5 to 8 points from A4 upward. Run in our host on the same 2,000 A10 games, Mentor's search wins 8.8% against 8.2% for our policies, which need 21-27 ms per decision on one CPU core. GPT-6 Astra at maximum reasoning won none of 11 A10 games through our text interface. In a controlled study of 18 training runs from six random initializations, failure-driven allocation exceeds uniform restart allocation on each of three Regent initializations, by 3.6 percentage points on average, while the Silent effect is small and changes sign. Code, trained policies, an in-game mod and the records behind every table are at https://github.com/HalfVulpes/sts2ai.
Figures & tables
| Character | Base | Released | Win rate (%) | 95% interval (%) | Change (pp) | |
|---|---|---|---|---|---|---|
| Ironclad | 317 | 399 | 15.59 | 14.23–17.04 | 3.20 | |
| Silent | 47 | 61 | 2.38 | 1.86–3.05 | 0.55 | .18 |
| Defect | 78 | 208 | 8.12 | 7.13–9.25 | 5.08 | |
| Necrobinder | 42 (unchanged) | 1.64 | 1.22–2.21 | |||
| Regent | 321 (unchanged) | 12.54 | 11.31–13.88 | |||
| Mean of five | 805 | 1031 | 8.05 | 1.77 | ||
| Character | A0 | A4 | A8 | A9 | A10 |
|---|---|---|---|---|---|
| Ironclad | 89.8 | 88.3 | 72.3 | 39.3 | 15.6 |
| Silent | 67.6 | 54.3 | 36.1 | 12.3 | 2.4 |
| Defect | 83.8 | 77.0 | 60.0 | 26.6 | 8.1 |
| Necrobinder | 72.7 | 59.4 | 34.2 | 5.9 | 1.6 |
| Regent | 87.1 | 78.5 | 56.4 | 30.1 | 12.5 |
| Mean of five | 80.2 | 71.5 | 51.8 | 22.8 | 8.1 |
| Agent | Computation per decision | Source | Games | Mean win (%) |
|---|---|---|---|---|
| Ours (released) | one network evaluation, 21–27 ms on one CPU core | measured | 12,800 | 8.05 |
| AlphaSpire networks without search [ 3 ] | one network evaluation | measured | 2,560 | 0.55 |
| Spire Mentor [ 8 ] | 8,000 MCTS simulations per combat decision, about 112 ms on one core | reported | 2,500 | 14.0 |
| Spire Mentor, in our host | 4,000–16,000 (Ironclad) or 8,000 simulations per combat decision | measured | 2,000 | 8.8 |
| GPT-6 Astra, our text interface | maximum reasoning, median 13.7 s | measured | 11 | 0.0 |
| DeepSeek V4.1 Flash 1017, our text interface | maximum reasoning, median 29 s | measured | 12 | 0.0 |
| Character | Ours | Interval | AlphaSpire | Interval |
|---|---|---|---|---|
| Ironclad | 75 | 11.85–17.97 | 14 | 1.64–4.54 |
| Silent | 9 | 0.93–3.31 | 0 | 0.00–0.74 |
| Defect | 47 | 6.97–11.99 | 0 | 0.00–0.74 |
| Necrobinder | 9 | 0.93–3.31 | 0 | 0.00–0.74 |
| Regent | 68 | 10.61–16.50 | 0 | 0.00–0.74 |
| Character | Mentor | Interval | Ours | Interval | Only Mentor / only ours ( ) |
|---|---|---|---|---|---|
| Ironclad | 67 | 13.4–20.7 | 53 | 10.3–16.9 | 44 / 30 (.13) |
| Regent | 18 | 2.9–7.0 | 55 | 10.7–17.5 | 16 / 53 ( .001) |
| Silent | 27 | 4.7–9.6 | 9 | 1.2–4.2 | 27 / 9 (.004) |
| Defect | 50 | 9.6–16.1 | 36 | 6.6–12.2 | 38 / 24 (.098) |
| Necrobinder | 14 | 2.1–5.8 | 10 | 1.4–4.5 | 13 / 9 (.52) |
| Mean of five | 8.8 | 8.2 |
| Character | Non-combat decisions | Mentor’s search in combat | Our policy in combat |
|---|---|---|---|
| Ironclad | Mentor | 67 | 22 |
| ours | 52 | 41 | |
| Regent | Mentor | 18 | 2 |
| ours | 70 | 55 |
| Factor | Setting |
|---|---|
| Methods | Fresh only; uniform situations + uncertainty; failure map + uncertainty |
| Task | Regent and Silent, A0, single player, no ascension curriculum |
| Budget | 763 updates of 65,536 rows; 50,003,968 rows per run |
| PPO | learning rate ; ; ; two epochs; minibatch 4,096 |
| Workers | 256 simulator processes |
| Development curves | EMA at updates 153, 306, 459, 611 and 763 on 256 development games |
| Character | Initialization | Fresh | Uniform | Failure | Failure uniform |
|---|---|---|---|---|---|
| Regent | 41001 | 74 | 54 | 99 | 4.39 |
| Regent | 41002 | 60 | 54 | 94 | 3.91 |
| Regent | 41003 | 80 | 106 | 131 | 2.44 |
| Silent | 52001 | 114 | 72 | 86 | 1.37 |
| Silent | 52002 | 28 | 53 | 40 | 1.27 |
| Silent | 52003 | 29 | 35 | 46 | 1.07 |
| Quantity | Fresh | Uniform | Failure |
|---|---|---|---|
| Optimizer rows (M) | 300.024 | 300.024 | 300.024 |
| Collected decisions (M) | 300.638 | 300.618 | 300.621 |
| Replayed prefix decisions (M) | 0 | 149.819 | 134.631 |
| Restart episodes | 0 | 591,062 | 577,858 |
| Completed episodes | 895,131 | 1,187,028 | 1,161,006 |
| Summed trainer hours | 15.17 | 17.41 | 17.88 |
| Character | Base | Selected | Difference (pp) | |
|---|---|---|---|---|
| Ironclad | 122 | 124 | 0.20 | .939 |
| Silent | 20 | 22 | 0.20 | .860 |
| Defect | 36 | 39 | 0.29 | .795 |
| Necrobinder | 15 | 15 | 0.00 | 1.000 |
| Regent | 135 | 141 | 0.59 | .705 |
| Character | Development | Test | Difference (pp) |
|---|---|---|---|
| Silent | 0.00 | ||
| Defect | 0.00 | ||
| Necrobinder | 0.49 | ||
| Ironclad | 0.68 | ||
| Regent | 2.93 |
| Character | Base | Trained | Discordant | Holm | Floor | A0 | A9 |
|---|---|---|---|---|---|---|---|
| Silent | 16 | 31 | 27 / 12 | .047 | 24.7 32.3 | 335 345 | 53 66 |
| Defect | 31 | 80 | 71 / 22 | 32.4 38.8 | 407 400 | 81 125 | |
| Necrobinder | 13 | 20 | 19 / 12 | .28 | 24.1 30.0 | 372 345 | 30 46 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter | Value | Parameter | Value |
|---|---|---|---|
| / GAE | 0.9998 / 0.99 | samples per iteration | 65,536 |
| clip (policy / value) | 0.2 / 0.2 | epochs, minibatch | 2, 4,096 |
| weight decay | 0.01 | gradient-norm clip | 1.0 |
| entropy (combat / other) | 0.003 / 0.01 | value / auxiliary coefficient | 0.5 / 0.2 |
| EMA decay per step | 0.998 | restart probability | 0.5 |
| situations | 12 | stored states per situation | 16 per worker |
| Continuation seed | Fresh | Uniform | Failure | Failure uniform |
|---|---|---|---|---|
| 101 | 151 | 141 | 153 | 6.0 pp |
| 202 | 132 | 144 | 137 | 3.5 pp |
| 303 | 130 | 126 | 134 | 4.0 pp |