Reinforcement Learning for the Game of Balatro: A Fast Simulator and Two-Policy Training Approach

Ajay Vikram Periasami; Anjaneya Gupta; Ethan Cheng; Ivan Rosales

cs.LGOct 28, 2026

Reinforcement Learning for the Game of Balatro: A Fast Simulator and Two-Policy Training Approach

Authors: Ajay Vikram Periasami, Anjaneya Gupta, Ethan Cheng, Ivan Rosales

Organizations: Department of Computer Science, Duke University, Durham, NC, USA

Abstract

Balatro, a 2024 roguelike deckbuilder, presents three difficulties that have historically derailed reinforcement learning methods: a 3,000-action state-dependent action space, horizons of several hundred decisions with sparse rewards, and pervasive stochasticity in both card draws and shop generation. Published RL results on the game remain scarce. We introduce an end-to-end pipeline comprising (i) a high-throughput Rust simulator that is automatically state-diffed against the live game client via BalatroBot, (ii) a condensed observation-action interface with dynamic per-step masking of legal moves, and (iii) a multi-agent PPO architecture that decomposes play into a tactical blind policy and a strategic shop policy that share the full game state but act in disjoint phases. Long-horizon credit assignment is partially addressed by per-agent reward routing with explicit cross-agent credit. On the main joint run, training metrics show a sharp phase transition in which the first-blind clear rate converges above 99% and mean episode return increases by roughly an order of magnitude, plateauing at a mean maximum ante of 3 out of 8; single-agent baselines under identical rewards oscillated between greedy and hoarding failure modes and did not stably clear the first blind. These results establish the two-policy decomposition as a strong reactive baseline for Balatro and localise the remaining gap to long-horizon strategic planning rather than tactical blind play.

Figures & tables

Panel of Figure 1 — Figure 1: The original Balatro game (left) and our web app implementation (right).

Figure 2: Multi-agent setup for blind play and shop decisions: shared observation preprocessing and a universal card encoder feed agent-specific backbones; the blind policy attends over hand (and joker) tokens before a joint trunk and heads, while the shop policy injects a projected global state into action tokens, applies attention, and merges token-level and abstract (meta-action) logits into the full discrete action space.

Hyperparameter	Value
Env-runner processes ( $\times$ envs per runner)	$4\times 1$
Train batch (env steps / iteration)	$5{,}120$
SGD minibatch size	$512$
SGD epochs per iteration	$4$
Rollout fragment length	auto ( truncate_episodes )
Discount $\gamma$	$0.99$

Table 1: PPO training hyperparameters used for the joint blind–shop run.

Panel of Figure 3 — Figure 3: Training metrics for the joint MARL run over the first 1000 PPO iterations. Curves are smoothed using a rolling mean with window size 40.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Range	Count	Action
0–9	10	SELECT_CARD_ $i$
10	1	PLAY_HAND
11	1	DISCARD_HAND
12–15	4	USE_CONSUMABLE_ $i$
16–25	10	BUY_ $i$ — shop slot $i$ , indexed to match shop_prices
26	1	REROLL_SHOP

Appendix

Table 2: V1 action space mapping (74 discrete actions).

Range	Count	Field
0–9	10	hand_cards
10–19	10	hand_enhancements
20–29	10	hand_editions
30–39	10	hand_seals
40–49	10	hand_debuff
50–59	10	selected

Appendix

Table 3: V1 observation space mapping (1449 dimensions). Left and right halves are contiguous; split here for typesetting.

RL for Game AI

See all 31 papers

CAST: Game Solvers as Turn-Level Teachers for LLM Agents
Jul 28, 2026Yu Wang, Yi-Kai Zhang, Wentao Shi +8Credit Assignment in RL Reinforcement Learning
University of Science and Technology of China · 4Meituan, China · 2Nanjing University +1
PokeRL: Reinforcement Learning for Pokemon Red
Apr 12, 2026Dheeraj Mudireddy, Sai PatibandlaRL for Game AI Reinforcement Learning
Texas A&M Institute of Data Science Texas A&M University College Station, USA · Dept. of Computer Science Texas A&M University College Station, USA
Causal Reinforcement Learning for Complex Card Games: A Magic The Gathering Benchmark
May 7, 2026Cristiano da Costa Cunha, Ajmal Mian, Tim French +1RL Benchmarks Structural Causal Models
Department of Computer Science and Software Engineering, University of Western Australia, 6009 WA Australia

Jul 28, 2026J/K move · Enter open · S save

CAST: Game Solvers as Turn-Level Teachers for LLM Agents

Yu Wang, Yi-Kai Zhang, Wentao Shi, Ziang Ye+7

University of Science and Technology of China · 4Meituan, China · 2Nanjing University+1

Credit Assignment in RL Reinforcement Learning Game-Playing Agents RL for LLM Agents Step-Level Credit Assignment in RL Reinforcement Learning with Verifiable Rewards+1

Open paper AskPDF arXiv

Training large language models (LLMs) to act in long-horizon games is a promising step toward generalist decision-making, yet reinforcement learning with verifiable rewards (RLVR) relies on sparse final rewards that reveal little about which decisions determine success. Denser process signals could supply this missing turn-level credit, but existing sources are hard to keep both cheap and accurate. We observe that changes in a game solver's state value reveal whether an action advances the state toward success. Building on this insight, we propose CAST (Credit Assignment from Solver Teachers), which converts these value changes into solver advantages and injects them into RLVR as turn-level signals. We further show that, under a soft-optimal solver assumption, maximizing the solver advantage is equivalent to on-policy distillation from the solver, requiring only scalar values rather than teacher logits. Across Sokoban, Minesweeper, and Rush Hour, CAST outperforms all trained baselines on every game under both in-domain and unseen-difficulty evaluation and achieves the highest average zero-shot performance on ALFWorld and WebShop. Our code is available at https://github.com/Wloner0809/CAST.

Abstract

RL for Game AI

Explore similar work

CAST: Game Solvers as Turn-Level Teachers for LLM Agents

PokeRL: Reinforcement Learning for Pokemon Red

Causal Reinforcement Learning for Complex Card Games: A Magic The Gathering Benchmark