Teaching AI Agent to Play Poker Strategically Through Preference Optimization

Phurinat Polasa; Thanapon Thanadunpremdet; Phatthanasak Kraiduang; Saksorn Ruangtanusak

cs.AIOct 19, 2026

Teaching AI Agent to Play Poker Strategically Through Preference Optimization

Authors: Phurinat Polasa, Thanapon Thanadunpremdet, Phatthanasak Kraiduang, Saksorn Ruangtanusak

Organizations: Department of Computer Engineering, KMUTT, Thailand · Department of Computer Engineering, Mahidol University, Thailand · R&D, SCB DataX, SCBX Group, Thailand

Abstract

A poker agent can make a legal move and still make a strategically poor one. We ask whether compact AI agents can exhibit strategic behavior under hidden information and whether preference optimization can improve beyond static and zero-shot policies. We first evaluate a static baseline and three frozen language models: SmallThinker, Qwen3, and Gemma-3 whose performance varies substantially between exact-game and six-player evaluation. We therefore propose Strategic Preference Optimization with KL-anchored Equilibrium (SPOKE), combining teacher-relative value penalties, policy anchoring, risk constraints, and structured rationale supervision, and compare it with SFT, DPO, GRPO, and GSPO. Across three runs of a SmallThinker-4B-A0.6B policy, SPOKE achieves the lowest mean exploitability in Kuhn (0.213) and Leduc (1.081) and the highest six-max top-1 agreement (0.775), while maintaining teacher KL comparable to SFT (0.522 vs 0.523). These results provide preliminary evidence that strategic preference optimization can improve strategic quality while preserving teacher-consistent behavior.

Figures & tables

Fig. 1: SPOKE overview: teaching a poker agent to act legally and strategically.

Panel of Figure 2 — Fig. 2: Proposed SPOKE pipeline. Teacher values are used only where validated; six-max action labels provide behavioral supervision. Legal-action filtering is shared by all methods.

Method	Model	Kuhn $\downarrow$	Leduc $\downarrow$	KL $\downarrow$	top-1 $\uparrow$
static	uniform / chart	0.458	2.374	1.104	0.643
zero-shot	SmallThinker-4B-A0.6B	0.326	1.767	1.865	0.159
zero-shot	Qwen3-0.6B	0.750	4.198	1.049	0.556
zero-shot	Gemma-3-1B-IT	0.833	6.999	4.834	0.214
SFT	SmallThinker-4B-A0.6B	0.276	2.210	0.523	0.754
DPO	SmallThinker-4B-A0.6B	0.758	3.666	5.480	0.174

TABLE I: Zero-shot controls vs. optimization methods (SmallThinker-4B-A0.6B). Three-run means; frozen controls are single runs. Bold: best per metric.

Game-Playing Agents

Game-playing agents use search, learning, and reasoning to choose actions in games with strategic, uncertain, or changing environments.

See all 202 papers

PokerSkill: LLMs Can Play Expert-Level Poker without Training or Solvers
May 28, 2026Boning Li, Baoxiang Wang, Longbo HuangGame-Playing Agents Imperfect-Information Games
IIIS, Tsinghua University · The Chinese University of Hong Kong, Shenzhen
AlphaExploitem: Going Beyond the Nash Equilibrium in Poker by Learning to Exploit Suboptimal Play
May 9, 2026Vlad Murgoci, Matthijs Spaan, Yaniv OrenOpponent Modeling Game-Playing Agents
Delft University of Technology · CD Delft, The Netherlands · Department of Intelligent Systems
Safe Equilibrium Policy Optimization for Strategic Agent Policies
May 29, 2026Karthika Arumugam, Kiran Kumar Manku, Amit DhandaMulti-Agent Reinforcement Learning Game-Playing Agents
†Amazon, USA

May 28, 2026J/K move · Enter open · S save

PokerSkill: LLMs Can Play Expert-Level Poker without Training or Solvers

Boning Li, Baoxiang Wang, Longbo Huang

IIIS, Tsinghua University · The Chinese University of Hong Kong, Shenzhen

Game-Playing Agents Imperfect-Information Games LLM Agents

Open paper AskPDF arXiv

Poker is a landmark challenge for artificial intelligence. The dominant approach relies on equilibrium solvers built on counterfactual regret minimization, requiring millions of core-hours of training. Large Language Models (LLMs) possess extensive poker knowledge but perform far below solver-based agents when asked to play directly. Traditional rule-based poker agents are interpretable and training-free, but their strategic ceiling remains far below equilibrium play. We introduce PokerSkill, a training-free and solver-free framework that bridges this gap by using detailed rule-based poker skills as a structured action-grounding interface for LLMs. A deterministic context engine analyzes the current state and retrieves only the relevant fragments from a layered skill library, which is entirely designed by human poker experts, constraining the LLM's choice to reasonable actions. Against GTOWizard, a state-of-the-art GTO benchmark, GPT-5.5 XHigh with PokerSkill achieves

-57 \pm 21

mbb/hand, Claude Opus 4.6 achieves

-80 \pm 29

mbb/hand and Claude Opus 4.7 achieves

-87\pm 64

mbb/hand, reducing losses by 49--61% compared to default-prompt baselines and outperforming the strong bot Slumbot. Our key finding is that rule-based skills alone do not constitute a strong strategy, and LLMs alone cannot play well, but their combination yields an agent that requires neither training nor solver access yet competes with systems built on millions of core-hours of computation. To our knowledge, this is the first demonstration of an LLM achieving competitive performance in a complex imperfect-information game without game-specific training or solver queries. Code is available at https://github.com/lbn187/PokerSkill.

Abstract

Game-Playing Agents

Explore similar work

PokerSkill: LLMs Can Play Expert-Level Poker without Training or Solvers

AlphaExploitem: Going Beyond the Nash Equilibrium in Poker by Learning to Exploit Suboptimal Play

Safe Equilibrium Policy Optimization for Strategic Agent Policies