Text Is the Bottleneck: A Small-Scale Replication of LLM Game Playing in Riichi Mahjong

Mingge Xu

cs.CLOct 26, 2026

Text Is the Bottleneck: A Small-Scale Replication of LLM Game Playing in Riichi Mahjong

Authors: Mingge Xu

Organizations: Department of Bioengineering, Graduate School of Engineering, The University of Tokyo.

Abstract

We replicate the riichi-mahjong track of "Can Large Language Models Master Complex Card Games?" (arXiv:2509.01328) under a drastically smaller budget: models of at most 2B parameters, one 16 GB consumer GPU, about five GPU-days. (1) With matched data and labels, text is the bottleneck: on the same 100k-decision training set, a 2.6M-parameter convolutional board encoder agrees with the strong AI Mortal on 80.6% of test decisions, above every text-prompt LLM we train (best 77.8%), and places better at the table (2.85 over 1,000 games vs. ~3.0 for the LLMs; paired difference at n=100 not significant). Scaled to 1.71M decisions, a 4.9M-parameter Transformer reaches 94.7% agreement, beats the 0.8B LLMs on identical seeds (-0.45 ranks, sign test p=0.005), and is statistically equivalent to parity against three Mortal champions (TOST within 0.10 ranks, p=6e-4); parity against a teacher-opponent is roughly the ceiling of pure imitation. (2) Training toward the teacher raises teacher agreement more than a 2.5x parameter increase does (+2.5 vs. +1.5 points); on human-record agreement both effects are +1.0, and neither is visible at the table. (3) Few-point agreement differences do not resolve into playing strength within a family: +2.5 agreement points coincide with a paired placement difference of +0.02 +/- 0.12 ranks (p=1.0), and +2.0 points at 2B with -0.06 (p=0.37) - both below the ~0.25-rank resolution of 100 paired games. We also document a paper-code mismatch in the original winner filter, reconstruct its two private engine patches, and show that a divide-and-conquer pipeline makes template-generated Japanese rationales 93% sentence-exact (43% without facts). Per Tenhou's terms we release no trained weights - only code, configurations, and evaluation protocols.

Figures & tables

Text-prompt LLMs (LoRA, 100k decisions unless noted)
Model	Supervision	H (%)	M (%)
human record (reference)	—	100	81.9
Qwen2.5-0.5B	human	68.5	71.4
TinySwallow-1.5B (20k)	human	59.9	61.1
Qwen3.5-0.8B	human	70.6	74.3
Qwen3.5-0.8B	Mortal labels	71.6	76.8

Table 1: Offline results on the unified 5,000-decision test set. Agreement is with the human record (H) or with Mortal’s first choice (M). All LLMs use text prompts; the encoder consumes a

58\times 34

plane + 71-scalar tensor computed from the same prompt. “Hard” = Mortal’s first choice only (identical supervision to the text LLMs, plus label smoothing 0.05); “soft” = additionally distilling Mortal’s Q-value distribution. TinySwallow is trained on 20k decisions (memory-limited) and listed for reference.

Figure 1: Agreement with Mortal’s first choice vs. parameter count (log scale). Text-prompt LLMs (circles: human labels; squares: Mortal labels) reach 71–78% across the 0.5B–2B range (TS-1.5B, 61%, was trained on 20k decisions—its 100k run was interrupted at step 5,005/6,188 and is not evaluated), while board encoders (diamonds) consuming the same prompt fields as tensors reach 80.6–94.7%; the dashed line marks human houou experts (81.9% on this test set). Encoder annotations give training-set size in decisions.

Agent	Placement dist.	Avg. rank	1st rate	Avg. pt
random legal action	$[0,1,8,91]$	3.90 $\pm$ 0.03	0%	$-122$
Qwen3.5-0.8B, human labels	$[10,16,35,39]$	3.03 $\pm$ 0.10	10%	$-36$
Qwen3.5-0.8B, Mortal labels (block A)	$[7,18,23,52]$	3.20 $\pm$ 0.10	7%	$-56$
Qwen3.5-0.8B, Mortal labels (block B, paired w. above)	$[10,23,23,44]$	3.01 $\pm$ 0.10	10%	$-40$
Qwen3.5-2B, human labels	$[14,16,25,45]$	3.01 $\pm$ 0.11	14%	$-41$
Qwen3.5-2B, Mortal labels (paired w. 2B-human)	$[14,17,29,40]$	2.95 $\pm$ 0.11	14%	$-34$

Table 2: Game-play results: one agent vs. three Mortal (VoidShine-298k) champions. 100 games each (25 seeds

\times

4 seat rotations), greedy decoding. Placement distribution

[1^{\text{st}},2^{\text{nd}},3^{\text{rd}},4^{\text{th}}]

, average placement

\pm

s.e., average Tenhou-style points (

90/45/0/-135

). Parity with the champion pool is 2.50. Our primary metric is average placement (the original paper’s protocol); the points column can rank differently because a 4th place costs

-135

(e.g., the 2B model’s extra first places do not offset its extra last places). Encoder rows are 1,000 games (250 seeds

\times

4 rotations, distributions as percentages) on the disjoint 95k seed block with seed-clustered s.e.; all other rows are 100 games. The encoders’ 100-game blocks (91k/92k) give 2.55/2.56 (TF) and 2.90/2.96 (CNN), consistent with the 1,000-game values.

Figure 2: Game-play evaluation vs. three Mortal champions (100 games per bar unless noted). (a) Placement distributions with average ranks above the bars; hatched bars are board encoders; dashed lines mark the cumulative placement boundaries under parity (25/50/75%). (b) Offline Mortal-agreement vs. average placement (error bars:

\pm

1 s.e.; H = human labels, M = Mortal labels pooled over two seed blocks): across representation families agreement tracks strength; within the text-LLM family few-point agreement differences do not resolve into placement differences (paired

+0.02\pm 0.12

ranks for

+2.5

agreement points).

Figure 3: The divide-and-conquer pipeline. A small perception head turns pixels into the structured state; a frozen CNN encoder compresses it into 35 soft tokens; a program appends checkable facts (including the encoder’s ranking in the rationale setting); the LLM integrates them. The percentages shown are from the clean-state pipeline: action 91.6% without facts, rationale sentences 92.8% with facts. The dashed path is the original paper’s text-only baseline.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

LoRA rank / target	8 (default) / all linear layers
template	qwen3_5_nothink (assistant answers directly; no think block)
epochs / steps	1 / 6,188 (100k decisions, effective batch 16)
optimizer / lr	AdamW / $10^{-4}$ cosine, 10% warmup
length / precision	4,096 tokens / bf16 + liger kernels [ 3 ]

Appendix

Table 3: Common LLM SFT configuration (LLaMA-Factory).

Game-Playing Agents

Game-playing agents use search, learning, and reasoning to choose actions in games with strategic, uncertain, or changing environments.

See all 201 papers

PokerSkill: LLMs Can Play Expert-Level Poker without Training or Solvers
May 28, 2026Boning Li, Baoxiang Wang, Longbo HuangGame-Playing Agents Imperfect-Information Games
IIIS, Tsinghua University · The Chinese University of Hong Kong, Shenzhen
Mahjax: A GPU-Accelerated Mahjong Simulator for Reinforcement Learning in JAX
May 20, 2026Soichiro Nishimori, Shinri Okano, Keigo Habara +3RL Benchmarks Game-Playing Agents
The University of Tokyo, Tokyo, Japan. · Nara Institute of Science and Technology, Nara, Japan. · Independent Researcher. +5
A Gold-Standard Study of What Makes a Lightweight Game-Playing Agent Strong
Jul 7, 2026Nima Kelidari, Mohammadsaeed Haghi, Mahdi SalmaniSelf-Play RL Reinforcement Learning
University of Southern California

May 28, 2026J/K move · Enter open · S save

PokerSkill: LLMs Can Play Expert-Level Poker without Training or Solvers

Boning Li, Baoxiang Wang, Longbo Huang

IIIS, Tsinghua University · The Chinese University of Hong Kong, Shenzhen

Game-Playing Agents Imperfect-Information Games LLM Agents

Open paper AskPDF arXiv

Poker is a landmark challenge for artificial intelligence. The dominant approach relies on equilibrium solvers built on counterfactual regret minimization, requiring millions of core-hours of training. Large Language Models (LLMs) possess extensive poker knowledge but perform far below solver-based agents when asked to play directly. Traditional rule-based poker agents are interpretable and training-free, but their strategic ceiling remains far below equilibrium play. We introduce PokerSkill, a training-free and solver-free framework that bridges this gap by using detailed rule-based poker skills as a structured action-grounding interface for LLMs. A deterministic context engine analyzes the current state and retrieves only the relevant fragments from a layered skill library, which is entirely designed by human poker experts, constraining the LLM's choice to reasonable actions. Against GTOWizard, a state-of-the-art GTO benchmark, GPT-5.5 XHigh with PokerSkill achieves

-57 \pm 21

mbb/hand, Claude Opus 4.6 achieves

-80 \pm 29

mbb/hand and Claude Opus 4.7 achieves

-87\pm 64

mbb/hand, reducing losses by 49--61% compared to default-prompt baselines and outperforming the strong bot Slumbot. Our key finding is that rule-based skills alone do not constitute a strong strategy, and LLMs alone cannot play well, but their combination yields an agent that requires neither training nor solver access yet competes with systems built on millions of core-hours of computation. To our knowledge, this is the first demonstration of an LLM achieving competitive performance in a complex imperfect-information game without game-specific training or solver queries. Code is available at https://github.com/lbn187/PokerSkill.

Abstract

Game-Playing Agents

Explore similar work

PokerSkill: LLMs Can Play Expert-Level Poker without Training or Solvers

Mahjax: A GPU-Accelerated Mahjong Simulator for Reinforcement Learning in JAX

A Gold-Standard Study of What Makes a Lightweight Game-Playing Agent Strong