Build, Don't Play: Self-Improving LLM Research Agents Develop a Hearthstone Battlegrounds AI That Rivals Top Players

Changlian Ma; Limin Wang

cs.AIOct 24, 2026

Build, Don't Play: Self-Improving LLM Research Agents Develop a Hearthstone Battlegrounds AI That Rivals Top Players

Authors: Changlian Ma, Limin Wang

Organizations: Nanjing University · Shanghai AI Laboratory

Abstract

Can LLM agents carry a hard real-world problem from raw data to a system that holds its own against people? We study this in Hearthstone Battlegrounds, an eight-player auto-battler with stochastic shops and battles, hidden information and nearly a thousand interacting cards. We present AutoTavern, a self-improving research system: specialised LLM research agents share a typed research memory, every claim passes verification gates, and live games close the loop, so the agents improve the simulator, opponents and policy and, from the same feedback, their own research process; a human sets goals and rules on ambiguous mechanics. In 20 days AutoTavern built TavernGym, the first faithful, full-content reproduction of the live game, and delivered 23 versions of TavernAgent, trained with PPO against opponents calibrated to real games. TavernAgent plays complete games on the live ladder by itself: its rating rose from 6,000 to about 15,000 and, on the public leaderboard's measure, its last 20 records average a placement of 1.60 with 80% first places, better than the server's ten highest-rated players on every statistic (2.34, 59%). TavernBench, 960 tasks in 40 scenarios, shows that the system the agents built is the strongest player: TavernAgent answers 91% of the play tasks against at most 63% for frontier LLMs, and decides in 0.03 s where they need up to 26 s; within the second a live turn allows, it keeps 90% and no LLM more than 26%. LLM agents build far better than they play. We will open-source TavernBench and TavernGym at https://github.com/MCG-NJU/AutoTavern.

Figures & tables

Figure 1: TavernBench at a glance. (a) Accuracy (%) per capability, with the play score counting only answers within the second a live turn allows per decision; numbers: TavernAgent . (b) Play score against seconds per decision; shaded: too slow for a live turn. All systems are in Table 4 .

Figure 2: Overview of AutoTavern , a self-improving research system. Six LLM research agents share one research memory and run three loops: training (minutes), research (hours) and the live ladder (days). The human sets goals, rules on ambiguous mechanics and presses the keys the AI names. Chart: first-place rate of the 23 delivered TavernAgent versions on one simulator build and three unseen seeds (24,576 games per point).

Duration (from scratch to ladder)	20 days	Experiments registered	about 10,000
Engine code (Rust)	63k lines	Delivered agent versions	23
Rule verification records	375	Fixes from live games	184
Human messages	418 (0.6% of all messages)

Table 1: Autonomous research in numbers.

	Full content, current patch	Heroes	Recruit + combat	Trainable batch API	Aligned with live games	Plays live games
MCTS assistant ( Zolboot et al., 2022 )	✗	partial	✓	✗	✗	✗
BGSimulator ( Elimelech, 2019 )	✗	✗	✓	✓	✗	✗
Firestone / Bob’s Buddy ( Zero-to-Heroes, 2017 ; HSReplay.net, 2020 )	✓	✓	✗	✗	✓	✗
DeepBattler ( William-Dic, 2025 )	n/a	n/a	n/a	✗	n/a	✗
TavernGym + TavernAgent (ours)	✓	✓	✓	✓	✓	✓

Table 2: Battlegrounds environments and agents. Only TavernGym reproduces the full live game, and only TavernAgent plays real games.

	Main exam			Harder exam
Agent	Place $\downarrow$	First %	Top-4 %	Place $\downarrow$	First %	Top-4 %
Random legal actions	7.97	0.0	0.0	7.97	0.0	0.0
Scripted bot	7.54	0.0	0.5	7.61	0.0	0.2
TavernAgent , first version (TA-1)	3.40	1.9	83.4	4.07	1.1	65.4
TavernAgent , mid version (TA-11)	2.83	18.5	85.5	3.75	8.9	67.8
TavernAgent , final (TA-23)	2.33	41.5	89.0	3.28	23.8	73.2

Table 3: TavernAgent against baselines, and the opponent ablation. One simulator build, three unseen seeds, 24,576 games per row, seven reality-calibrated opponent seats; main exam (left) and the harder exam (right). The ablation pair shares its starting point and training budget.

Figure 3: The climb on the live ladder. Rating after each of 392 games, days 5–18. Grey bands: the versions in play ( Table 10 ); orange: the two pauses in which the loop fixed what the ladder had exposed ( Section 8 ). Strip: one tick per game with its placement;

\times

: client failure.

System	Play	Real-time	Combat	Position	Order	Targets	Select	Strategy	Rules	Time (s)
TavernAgent	91	90	100	98	100	98	64	75	–	0.03
Claude Opus 5.5	52	0	67	32	60	52	41	38	92	6.2
Claude Sonnet 5.5	53	0	58	45	68	52	39	29	85	1.4
GPT-6 Astra	63	0	75	55	72	55	50	54	95	2.4
GPT-6.1 Sol	46	0	50	42	48	28	55	43	56	2.0
Gemini 3.8 Flash	44	0	52	32	32	48	41	43	90	9.9

Table 4: TavernBench results (368 tasks; accuracy in %). Play: the 30 scenarios TavernAgent answers, mean over scenarios. Real-time: the play score counting only answers given within 1 s. Rules: the ten rule-execution scenarios (answered by TavernGym by construction, so not scored for TavernAgent ). Time: median seconds per task (LLMs: estimated from Artificial Analysis medians, Appendix E ; others measured).

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Role	Responsibilities
Research lead	reads the memory, chooses the next problem, dispatches work, records findings
Environment	implements rules and content, verifies rules, fixes discrepancies found in live games
Training	designs recipes and curricula, registers training runs and their successors
Opponent modelling	builds and recalibrates the human-like opponent libraries
Evaluation & audit	evaluates within one build, re-checks candidates on unseen seeds, reviews
Deployment	maintains the client controller and the models’ input formats, collects live games

Appendix

Table 5: Roles in AutoTavern .

Figure 4: TavernGym . The Rust engine, its Python bindings and the programs built on them; the client controller feeds the live game to the agent in the same encoding.

Release	Day	Change	Release	Day	Change
TA-1	4	first model	TA-13	13	main line
TA-2	5	main line	TA-14	14	opponents recalibrated to live games
TA-3	6	opponents fitted to human statistics; first-place goal	TA-15	14	play-order fix
TA-4	6	main line	TA-16	14	new opponent libraries
TA-5	7	main line	TA-17	15	graded opponent libraries
TA-6	7	main line	TA-18	16	environment update

Appendix

Table 6: TavernAgent releases. TA-1 to TA-23 in release order, by day of the project, with the change each release introduced; the paper uses these names throughout.

Figure 5: TavernAgent ’s network. Entity tokens, a card representation that includes the card’s rule text, a transformer trunk and three heads.

Games per member $\times$ steps	512 $\times$ 256	Iterations per stage	300
Decisions per iteration	about $1.05\times 10^{6}$	Minibatch / epochs	8,192 / 1
Learning rate	$6\times 10^{-6}\to 5\times 10^{-6}$	Clip / target KL	0.2 / 0.05
$\gamma$ / $\lambda$	1.0 / 0.995	Gradient norm clip	1.0
Entropy / value / auxiliary coefficients	0.002 / 0.5 / 0.1	KL anchor to the stage’s start	0.1 for 100 iterations

Appendix

Table 7: Training configuration of the final stage (TA-23).

	Main exam			Harder exam
Network	Place $\downarrow$	First %	Top-4 %	Place $\downarrow$	First %	Top-4 %
3.5M parameters (d=192, 4 layers)	3.84	0.3	73.9	4.47	0.3	54.0
8M parameters (d=256, 6 layers)	3.77	0.5	75.6	4.40	0.3	56.2
18M parameters (d=384, 8 layers)	3.67	0.5	78.5	4.28	0.4	59.6

Appendix

Table 8: Model size (an early experiment: from scratch, same recipe, 400 iterations each).

Figure 6: Warband strength by turn. TavernAgent versions on the main exam (three seeds) and human players of the same patch (public statistics), log scale.

Version	Day	Avg. place	First (%)	Top-4 (%)
TA-23	18	3.95	26.2	57.9
TA-21	17	4.06	23.5	56.7
TA-17	15	4.36	19.7	51.6
TA-14	14	4.40	14.9	52.4
TA-11	10	4.53	9.4	51.9
TA-3	6	4.76	4.4	46.6

Appendix

Table 9: Eight versions in the same lobbies (three seeds, 24,576 games per version; a fair share is a placement of 4.5, 12.5% first and 50% top-four).

Figure 7: Heroes. Average placement of TavernAgent (TA-23) with each hero against human players’ average placement with that hero (public statistics). Dashed lines: the two means; the six heroes where the two disagree most are named.

Figure 8: Decision value. Average placement lost by TavernAgent (TA-23) when one class of its decisions is taken away (made at random, never made, or the information hidden); main exam, three seeds, 24,576 games per bar.

Versions	Days	Games	Avg. place	First (%)	Top-4 (%)	Rating
TA-2	5–6	26	3.46	3.8	76.9	6,000 $\to$ 7,177
TA-5–6	7–8	10	3.30	20.0	80.0	7,430 $\to$ 8,225
TA-8–10	8–10	47	3.57	12.8	66.0	8,117 $\to$ 8,348
TA-11	11–13	87	3.10	16.1	78.2	8,348 $\to$ 10,910
TA-12–14	13–16	28	3.36	7.1	78.6	11,001 $\to$ 12,329
TA-16–17	15–16	30	3.67	16.7	70.0	11,995 $\to$ 12,727

Appendix

Table 10: Live-ladder games by TavernAgent version (games linked to a single version with high confidence, without games with a client failure; ratings at the first and last game of each group). All games, failures included, are in Table 12 .

Player	Avg. place	First (%)	Top-4 (%)	Rating change
P1	3.03	50	65	$-$ 319
P2	2.20	70	80	+289
P3	2.43	55	85	$-$ 88
P4	1.88	65	90	+487
P5	2.50	55	75	$-$ 76
P6	2.05	70	85	+459

Appendix

Table 11: The agent against the server’s ten highest-rated players : each player’s last 20 records on the public leaderboard ( wallii.gg, 2026 ) at the cut-off (placements estimated by the leaderboard from rating changes, the same measure for every player; players anonymised).

Versions	Games	Avg. place	First (%)	Top-4 (%)	With failure
TA-2	40	3.38	2.5	77.5	14
TA-5–6	24	3.12	20.8	79.2	14
TA-8–10	68	4.22	8.8	54.4	21
TA-11	89	3.07	16.9	78.7	2
TA-12–14	38	3.08	5.3	84.2	10
TA-16–17	48	3.15	20.8	79.2	18

Appendix

Table 12: All live-ladder games by version , client failures included.

Category	Scenarios	Label	TavernAgent
Rules (7)	legal actions; one-on-one fights; battlecry, spell and magnetic results; end- and start-of-turn effects; triples	executed	one-on-one
Economy (5)	gold accounting; upgrade price; can I triple now; gold chains; best purchase for the next combat	executed / simulated	purchase
Combat (6)	who attacks first; deterministic fights; win / tie / loss; expected damage; lethal risk; start-of-combat effects	simulated	all
Positioning (5)	best of five arrangements; taunt and shield; deathrattle and reborn; against venomous; against cleave	simulated	all
Ordering (5)	sell timing; play triggers; spells and minions; adjacent battlecries; mixed	simulated	all
Targets (5)	spell, activate, Hero Power and magnetic targets; which minion to sell	simulated	all

Appendix

Table 13: TavernBench categories. 40 scenarios, 960 tasks. TavernAgent answers through its own interface ( Section 7 ); the ten rule-execution scenarios have no agent counterpart, because TavernGym executes the rules for it.

Category	Question	Label
Rules	You cast Misplaced Tea Set. List the Attack and Health of every minion afterwards.	[[9,6],[13,8]] (executed, three reseeded copies agree)
Economy	Which tavern minions do you buy, in order, with 9 Gold, each played at once, to win the next combat against the warband shown?	positions 2,1,4 (win rate 8.9%, best of 157 sequences)
Combat	Estimate the probabilities that you win, tie or lose this combat.	0.75 / 0.20 / 0.04 (4,096 combats)
Positioning	Which of five arrangements of your warband (against a Cleave opponent) wins most often?	E (best by $\geq$ 5 points over 2,048 combats each)
Ordering	In which order do you play three minions and sell one before the next combat?	B, D, C, A (win rate 90%)
Targets	Onto which of your Mechs do you magnetize Enchanted Sentinel?	position 0 (71%)

Appendix

Table 14: Example tasks , one per category (questions abridged; each task also shows the state and the rule text of every card). Labels come from TavernGym .

Model	Tasks	Text	Tools	Position	Order	Targets	Purchase	Select	Agent	Calls	Time (s)
GPT-6 Astra	144	62	97	55/98	72/100	55/92	75/88	69/100	99	5.2	21
GLM-5.3	144	38	81	20/82	40/62	38/95	38/62	75/94	99	5.8	11
Kimi-K2.6	144	26	65	8/68	22/35	38/98	25/0	50/88	99	8.9	39
Intern-S2-Preview	144	32	41	12/40	32/30	40/45	12/12	69/75	99	9.8	18
DeepSeek-V4.1-Flash	144	24	64	18/50	15/42	32/92	38/25	31/100	99	8.6	42

Appendix

Table 15: LLMs with TavernGym as a tool (accuracy in %, text-only / with tools per category; the agent column is TavernAgent on the same tasks).

LLM Agent Self-Improvement

Large language model agent self-improvement studies how agents use experience and feedback to improve their behavior, tools, and workflows.

See all 273 papers

PTCG-Bench: Can LLM Agents Master Pokémon Trading Card Game?
May 28, 2026Dongdong Hua, Yifei Sun, Renhong Huang +3Game-Playing Agents LLM Agent Evaluation
Zhejiang University · FinVolution Group
SpeedrunBench: Challenging LLM Agents with Video Game Speedrunning
Oct 6, 2026Yoshinari Fujinuma, Keisuke Kamahori, Ryuto Koike +8LLM Agent Evaluation Self-Improving Agents
Patronus AI · University of Washington · Institute of Science Tokyo
PostTrainBench: Can LLM Agents Automate LLM Post-Training?
Mar 9, 2026Ben Rank, Hardik Bhatnagar, Ameya Prabhu +4LLM Post-Training Autonomous Research Agents
ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems · Tübingen AI Center +2

May 28, 2026J/K move · Enter open · S save

PTCG-Bench: Can LLM Agents Master Pokémon Trading Card Game?

Dongdong Hua, Yifei Sun, Renhong Huang, Feng Gao+2

Zhejiang University · FinVolution Group

Game-Playing Agents LLM Agent Evaluation AI Agent Benchmarks Self-Evolving Agents

Open paper AskPDF arXiv

Given a strategically complex board game, human players can quickly learn to devise strategies after playing a few rounds. Autonomous agents require similar capabilities in realistic interactive environments, yet existing agent benchmarks often fail to fully capture such strategic and evolving decision-making scenarios. We present PTCG-Bench, a benchmark built on the Pok'{e}mon Trading Card Game (PTCG) that evaluates LLM agents at two complementary levels: (1) their decision-making performance within a single complex environment, and (2) their ability to self-evolving through accumulated experience. We further include a modular harness ablation to better interpret agent performance without conflating it with model capability. Our experiments show that, although LLM agents can achieve non-trivial gameplay performance, sustained and stable self-evolution remains challenging, and performance is sensitive to harness design. We hope that PTCG-Bench will facilitate future research on harness-aware and self-evolving agents in realistic interactive environments.

Abstract

LLM Agent Self-Improvement

Explore similar work

PTCG-Bench: Can LLM Agents Master Pokémon Trading Card Game?

SpeedrunBench: Challenging LLM Agents with Video Game Speedrunning

PostTrainBench: Can LLM Agents Automate LLM Post-Training?