cs.AIApr 5, 2026

Readable Minds: Emergent Theory-of-Mind-Like Behavior in LLM Poker Agents

Authors: Hsieh-Ting Lin, Tsung-Yu Hou

Organizations: Department of Oncology, Koo Foundation Sun Yat Sen Cancer Center, Taipei, Taiwan · Department of Digital Content and Technologies, National Chengchi University, Taipei, Taiwan

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Theory of Mind (ToM) -- the ability to model others' mental states -- is fundamental to human social cognition. Whether large language models (LLMs) can develop ToM has been tested exclusively through static vignettes, leaving open whether ToM-like reasoning can emerge through dynamic interaction. Here we report that autonomous LLM agents playing extended sessions of Texas Hold'em poker progressively develop sophisticated opponent models, but only when equipped with persistent memory. In a 2x2 factorial design crossing memory (present/absent) with domain knowledge (present/absent), each with five replications (N = 20 experiments, ~6,000 agent-hand observations), we find that memory is both necessary and sufficient for ToM-like behavior emergence (Cliff's delta = 1.0, p = 0.008). Agents with memory reach ToM Level 3-5 (predictive to recursive modeling), while agents without memory remain at Level 0 across all replications. Strategic deception grounded in opponent models occurs exclusively in memory-equipped conditions (Fisher's exact p < 0.001). Domain expertise does not gate ToM-like behavior emergence but enhances its application: agents without poker knowledge develop equivalent ToM levels but less precise deception (p = 0.004). Agents with ToM deviate from game-theoretically optimal play (67% vs. 79% TAG adherence, delta = -1.0, p = 0.008) to exploit specific opponents, mirroring expert human play. All mental models are expressed in natural language and directly readable, providing a transparent window into AI social cognition. Cross-model validation with GPT-4o yields weighted Cohen's kappa = 0.81 (almost perfect agreement). These findings demonstrate that functional ToM-like behavior can emerge from interaction dynamics alone, without explicit training or prompting, with implications for understanding artificial social intelligence and biological social cognition.

Figures & tables

Figure 1: Real-time spectator interface during a Full condition session (Hand 3, Turn). The interface displays each agent’s cards, chips, and position around the poker table. The Theory of Mind panel (top left) shows real-time ToM level assessments for each agent. Agent memory notes appear as speech bubbles, illustrating the natural-language opponent models that constitute “readable minds.” Statistics panel (left) tracks VPIP, PFR, aggression factor, and win rate across the session.
Figure 2: ToM level trajectories across hands. Mean maximum ToM level per hand bin, averaged across five replications. Memory-equipped conditions (Full, No-Skill) show progressive development from Level 0 to Level 3–5, while memory-absent conditions (No-Memory, Baseline) remain at Level 0 throughout. Shaded regions indicate ±\pm 1 SD across replications.
Figure 3: TAG adherence and chip spread by condition. (A) Mean TAG adherence (preflop action alignment with tight-aggressive baseline strategy) across the four conditions. Memory-equipped conditions show significantly lower TAG adherence, reflecting deliberate exploitation of opponent tendencies. (B) Final chip spread (richest minus poorest player) at session end. Error bars indicate ±\pm 1 SD across five replications.
ConditionMemorySkillMax ToMTAG Adh.DeceptionChip Spread
   Level(%)(Tier 2, %)(chips)
FullYesYes3.0±0.03.0\pm 0.067±467\pm 418.245,246±19,70245{,}246\pm 19{,}702
No-SkillYesNo2.8±0.452.8\pm 0.4568±568\pm 511.638,104±21,34838{,}104\pm 21{,}348
No-MemoryNoYes0.0±0.00.0\pm 0.079±379\pm 3022,535±23,14722{,}535\pm 23{,}147
BaselineNoNo0.0±0.00.0\pm 0.078±478\pm 4021,891±18,53221{,}891\pm 18{,}532
Table 1: Key metrics by condition. Memory = persistent memory file present; Skill = poker strategy guidance in system prompt; Max ToM Level = maximum ToM level reached (mean ±\pm SD across 5 replications); TAG Adherence = mean preflop TAG adherence (alignment with tight-aggressive baseline strategy); Deception (Tier 2) = proportion of hands with ToM-grounded deception; Chip Spread = mean final chip spread (richest minus poorest player).
ConditionTier 1 Rate (%)Tier 2 Rate (%)
Full25.018.2
No-Skill16.411.6
No-Memory14.00
Baseline19.80
Table 2: Deception rates by condition. Tier 1 = simple bluffs (raising with weak hands regardless of opponent model); Tier 2 = ToM-grounded deception (bluffing specifically because the agent’s model predicts the opponent will fold). Rates are proportion of hands containing at least one deception event, pooled across five replications.
Figure 4: Cross-model inter-rater reliability confusion matrix. Comparison of ToM level codes assigned by Claude Sonnet (rows) and GPT-4o (columns) on a stratified random sample of memory snapshots. The concentration of values on the main diagonal reflects near-perfect agreement ( κ=0.81\kappa=0.81 ). All disagreements are between adjacent levels, confirming systematic consistency in the coding rubric.
More in this topic

Theory of Mind

Theory of Mind studies how people and AI infer others’ beliefs, goals, and knowledge to predict behavior and coordinate interaction.

See all 92 papers

Explore similar work

CardsList
May 28, 2026

PokerSkill: LLMs Can Play Expert-Level Poker without Training or Solvers

Poker is a landmark challenge for artificial intelligence. The dominant approach relies on equilibrium solvers built on counterfactual regret minimization, requiring millions of core-hours of training. Large Language Models (LLMs) possess extensive poker knowledge but perform far below solver-based agents when asked to play directly. Traditional rule-based poker agents are interpretable and training-free, but their strategic ceiling remains far below equilibrium play. We introduce PokerSkill, a training-free and solver-free framework that bridges this gap by using detailed rule-based poker skills as a structured action-grounding interface for LLMs. A deterministic context engine analyzes the current state and retrieves only the relevant fragments from a layered skill library, which is entirely designed by human poker experts, constraining the LLM's choice to reasonable actions. Against GTOWizard, a state-of-the-art GTO benchmark, GPT-5.5 XHigh with PokerSkill achieves −57±21-57 \pm 21 mbb/hand, Claude Opus 4.6 achieves −80±29-80 \pm 29 mbb/hand and Claude Opus 4.7 achieves −87±64-87\pm 64 mbb/hand, reducing losses by 49--61% compared to default-prompt baselines and outperforming the strong bot Slumbot. Our key finding is that rule-based skills alone do not constitute a strong strategy, and LLMs alone cannot play well, but their combination yields an agent that requires neither training nor solver access yet competes with systems built on millions of core-hours of computation. To our knowledge, this is the first demonstration of an LLM achieving competitive performance in a complex imperfect-information game without game-specific training or solver queries. Code is available at https://github.com/lbn187/PokerSkill.

Boning Li, Baoxiang Wang, Longbo Huang

IIIS, Tsinghua University · The Chinese University of Hong Kong, Shenzhen

Game-Playing AgentsImperfect-Information GamesLLM Agents
AskPDF
Aug 10, 2026

Avalon-ToM-Bench: Evaluating Fine-Grained Theory of Mind via Asymmetric Game Mechanics

Theory of Mind (ToM) is essential for agent interactions, yet existing evaluations either rely on static scenarios that oversimplify mental-state reasoning or interactive settings that provide limited diagnostic insight. We present Avalon-ToM-Bench, a fine-grained benchmark that operationalizes ToM through the asymmetric-information mechanics of The Resistance: Avalon. Rather than evaluating end-to-end gameplay, it decomposes ToM into a 2×\times2 taxonomy -- epistemic versus motivational reasoning crossed with inference versus action -- using human-crafted, perspective-constrained queries. Benchmarking 28 LLMs reveals three insights: 1) Reasoning, not knowledge. Models show strong game-rule comprehension but markedly weaker ToM abilities, isolating failures to social reasoning rather than missing domain knowledge. 2) Expression, not representation. Mechanistic analyses via linear probing and activation steering show that models frequently represent correct mental-state inferences in their hidden states but fail to express them during generation -- linear probes recover 77-82% accuracy versus 62-70% from the models' own chain-of-thought. 3) Policy, not deliberation. Dedicated reasoning training yields substantial improvements whereas test-time chain-of-thought provides only marginal gains (+11.0 versus +1.1 points on average), suggesting that robust ToM depends on a learned reasoning policy rather than increased inference-time deliberation.

Yen-Shan Chen, Yu Chian Duan, Chih-En Kuo +2

1National Taiwan University · 2CyCraft AI Lab, Taiwan

LLM EvaluationTheory of MindLLM Reasoning
AskPDF
Mar 27, 2026

Selective Deficits in LLM Mental Self-Modeling in a Behavior-Based Test of Theory of Mind

The ability to represent oneself and others as agents with knowledge, intentions, and belief states that guide their behavior - Theory of Mind - is a human universal that enables us to navigate - and manipulate - the social world. It is supported by our ability to form mental models of ourselves and others. Its ubiquity in human affairs entails that LLMs have seen innumerable examples of it in their training data and therefore may have learned to mimic it, but whether they have actually learned causal models that they can deploy in arbitrary settings is unclear. We therefore develop a novel experimental paradigm that requires that subjects form representations of the mental states of themselves and others and act on them strategically rather than merely describe them. We test a wide range of leading open and closed source LLMs released since 2024, as well as human subjects, on this paradigm. We find that 1) LLMs released before mid-2025 fail at all of our tasks, 2) more recent LLMs achieve human-level performance on modeling the cognitive states of others, and 3) even frontier LLMs fail at our self-modeling task - unless afforded a scratchpad in the form of a reasoning trace. We further demonstrate cognitive load effects on other-modeling tasks, offering suggestive evidence that LLMs are using something akin to limited-capacity working memory to hold these mental representations in mind during a single forward pass. Finally, we explore the mechanisms by which reasoning models succeed at the self- and other-modeling tasks, and show that they readily engage in strategic deception.

Christopher Ackerman

Working Memory in Language ModelsTheory of MindMetacognition in Language Models
AskPDF