cs.AIOct 12, 2026

Beyond Successful Play: Diagnosing Opponent Modeling in Large Language Models

Authors: Hanbo Xie, Yue Zhao, Robert C. Wilson

Organizations: University of Sussex

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Language-model agents are expected to adapt to others whose behavior changes through interaction. Yet adaptation may rely on learned behavioral patterns rather than theory of mind (ToM): the ability to infer others' mental states, including beliefs and intentions. We introduce a benchmark and diagnostic framework for repeated hide-and-seek against algorithmic opponents with explicitly defined recursive belief updates. Their ground-truth beliefs and action probabilities let us distinguish adaptive play, action prediction, and belief tracking through live interaction, fixed-history evaluation, and controlled information interventions. With structured elicitation and behavioral supervised fine-tuning (SFT), we find improved play in several tested settings, but diagnostics reveal limitations in opponent prediction and belief tracking. We test whether supervising opponent action probabilities can address this predictive deficit in Qwen2.5-7B-Instruct. Predictions improve when the model is told the opponent's type and parameter settings, but become less accurate when it must infer the opponent from history without this information. Training on examples that also omit these details reduces the gap, although improvements vary across opponents. Is this limitation specific to smaller open-weight models, or does it also appear in a frontier model? GPT-5.4 also benefits from correct opponent beliefs: supplying them improves its predictions and choices, whereas deeper explicit belief reports and self-generated memory do not consistently improve forecasting from interaction history. This contrast suggests that using useful belief information is easier than inferring it autonomously, even for a frontier model. Together, our findings distinguish improved play and action prediction from reliable belief tracking, providing a framework for evaluating mentalizing and testing targeted improvements.

Figures & tables

Figure 1: From static ToM evaluation to dynamic opponent modeling. (A) Static evaluations ask models to answer belief questions about a fixed narrative. (B) Repeated interaction requires anticipating an adaptive opponent while one’s own actions shape subsequent observations. (C) Our evaluation distinguishes opponent action prediction, role-conditioned response, and payoff. The dashed pathway denotes opponent-state inference as a process to investigate, not an established internal mechanism. Explicit prediction is required only in designated evaluation interfaces, not in natural online play.
Figure 2: Live play across conditions. Blue denotes seeker and orange hider, including in (c); circles/squares distinguish action-only/structured output, with line style also used in (e). In (d), circles/squares indicate before/after SFT. (a) Qwen2.5 size and (b) reasoning settings against ToM2. Gemma standard hider uses an audited historical action-only run; Gemma thinking and GPT-OSS hider settings use audited replacements, not paired controls. (c) Output-format comparisons by role. (d) Qwen2.5-7B structured-output behavioral supervised fine-tuning (SFT) against ToM2 and held-out ToM3; (e) action-only and structured-output behavioral SFT transfer against GPT-5.4. Each completed cell has 20 live matches of 50 rounds. Whiskers show 95% match-bootstrap CIs where available; panel c omits intervals because archived seeker CIs cannot all be verified. Prespecified random fallback occurred in 30 Gemma thinking rounds and 3/4/10 GPT-OSS low/medium/high hider rounds; all rounds remain in the results. Missing cells are not zero.
Figure 3: Prediction learning and belief reporting are distinct. Qwen2.5-7B fixed-history studies. Colors distinguish training conditions, not roles: both roles are pooled except in (f), which separates them by row. (a,b) Action confidence and belief reports on six matches. (c) SFT on qtq_{t} alone versus a numeric state vector and qtq_{t} , with opponent specification, 60 matches; invalid outputs receive loss 1. (d) Five-field value SFT, including belief targets, on six matches. “Initial” marks the start of each five-field run; one run starts from the initial model and the other from a behavioral-SFT checkpoint. (e,f) Equal-update probability-only SFT continuations on a fresh R5 cohort, with specification on all versus half of training examples; (e) action-probability MSE by test input and (f) history-only MSE differences by opponent. Both use 60 fresh matches and 540 noninitial predictions per condition. Dashed lines mark constant baselines; (f) shows 95% paired match-bootstrap intervals. Panels use distinct cohorts and interfaces (Appendix E ).
Figure 4: Using supplied opponent information differs from inferring it. GPT-5.4 medium on fixed histories. Colors encode conditions, not roles; only (b) separates them. (a) Known-rule execution, six matches; incomplete chains count as errors. (b,c) No-rule replication: 24 matches, 216 noninitial points per arm; the supervised control has additional task-specific training. In the six-match supplied-belief intervention, (d) plots forecast MSE and (e) plots expected-regret differences across 54 decisions per arm. The 66.7% to 96.3% reference-best-action agreement is a separate audit summary of these decisions (Appendix F ), not a plotted axis. Regret whiskers show paired 95% match-bootstrap CIs. (f) Six-match report-depth pilot: forecast MSE minus constant, without intervals. These studies are separate; Appendix F details retry-assisted coverage.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

StudyInput beyond task/role/historyRequested output
Natural liveNone; unidentified opponentAction
Structured liveCandidate descriptions vary by archived promptStrategy/action chain and final action
Qwen token probeNoneDigit 0/1, scored via normalized logits
Qwen belief probeNoneDecimal opponent belief bb
Probability-only SFTOpponent specificationDecimal action probability qq
State-plus-probability SFTSpecification; true prior in warmup, own prior in evaluationNumeric state vector and qq
Appendix
Table 1: Information and output contracts. “Own state” always means previously generated state unless teacher forcing is explicitly indicated.
Figure S1: Live play across opponents. Mean 50-round payoff by role, from 20 matches per cell. Purple denotes negative payoff and teal positive payoff in both panels; color does not encode role or significance. The reasoning block shows Gemma 4 thinking and GPT-OSS low, medium and high effort across all six opponents. It combines historical seeker runs with audited hider replacements, rather than paired role or effort comparisons. Prespecified random fallback actions remain in the reported payoffs. The Qwen2.5-72B row is repeated in the scaling and flagship groups using the same source cells. Qwen3.6 uses no thinking.
Figure S2: Structured-output payoff changes. Each value is structured minus action-only payoff per 50-round live match; purple denotes a negative difference and teal a positive difference in both panels. Color is not a significance test. The scaling block covers seven Qwen2.5 sizes from 0.5B to 72B, both roles and all six opponents. The 72B row is repeated in the flagship group using the same contrasts. All 48 added structured-output cells have 20 complete matches; 0.5B scores retain 1,184 prespecified random fallback actions among 12,000 decisions. GPT-5.4 controls are matched by simulator seed; open-model controls are unpaired historical runs and may differ in prompt, runtime or budget.
Figure S3: Behavioral SFT across six opponents and both roles. Qwen2.5-7B, 20 live 50-round matches per cell. Blue denotes seeker and orange hider throughout; circles are the untrained interface baseline and squares the corresponding SFT checkpoint. Whiskers are archived 95% whole-match bootstrap CIs. Action-only step 2400 and structured-output step 300 are separate training packages, each compared with its corresponding baseline; they are not equal-update conditions. These evaluations are live play, not training-history replay. ToM3 and GPT-5.4 transfer are distinct evaluations in Figure 2 d,e.
Figure S4: Offline action-chain GRPO across opponents. Each cell gives mean payoff per 50-round live match, with 20 matches per role and checkpoint. Teal denotes positive payoff and purple negative payoff in both role panels; numbers give the exact archived means. All nine algorithmic opponents and GPT-5.4 are retained. No matched pre-RL checkpoint or uniform uncertainty layer is available, so color differences are descriptive, not a significance test.
Figure S5: Iterative GRPO against ToM2. Points are mean 50-round payoff averaged across seeker and hider, each evaluated in 20 matches at the SFT start and each iteration endpoint. Teal circles and purple squares distinguish K=4K=4 and K=16K=16 ; color does not encode role. Lines join checkpoints within a training run; evaluation cohorts differ between group sizes. No comparable confidence interval is available for every point, and these curves do not establish a reliable update or group-size effect.
Figure S6: External-prefix diagnostic on fixed histories. Each arm has 600 decisions from 12 histories. Teal circles identify structured SFT and purple squares recursive GRPO; color does not encode role. Left: action-probability MSE (lower is better); right: agreement with the reference best action (higher is better), not live win rate. Text labels retain the archived point estimates. Supplying true qq fixes that scored field, so its near-zero MSE is not a learned forecast. No arm-level uncertainty is plotted; the paired true-versus-shuffled-belief interval is reported in the text.
More in this topic

Opponent Modeling

Opponent modeling studies how agents infer others’ goals, preferences, policies, and hidden states to predict and adapt to their actions.

See all 17 papers

Explore similar work

CardsList
Aug 26, 2026

Assessing mentalization in humans and large language models

Mentalization - the ability to infer others' beliefs and intentions to guide one's own choices - is a key cognitive function underlying human social interactions. Large language models (LLMs) demonstrate behaviour consistent with humans on theory-of-mind tasks, yet whether these models can guide adaptive behaviour through mentalization is unknown. Here we use two economic games with cognitive computational modeling to uncover the latent strategies underlying mentalization in LLMs. We tested individual LLM agents across four model families, DeepSeek, GPT-4.1, GPT-5 and Gemini 2.0 Flash (N = 2,099), against opponents of varying sophistication and examined whether a prompting strategy designed to elicit strategic reasoning improved performance. We benchmarked results against human participants (N = 251) as a comparative measure. Across both games, LLMs showed clear behavioural and computational signatures of mentalizing that differed markedly by model provider and size. Strategic prompting generally improved performance by inducing more sophisticated reasoning, yet the extent of the benefit differed across the two tasks. Last, GPT-5 agents flexibly adapted their recursive depth of reasoning to increasingly sophisticated opponents, demonstrating superior performance to human participants. Collectively, we demonstrate different capacities for mentalization across LLMs, and highlight cognitive computational modeling as a formal method for assessing comparative intelligence across humans and machines.

Aamir Sohail, Xintong Zhong, Arkady Konovalov +2

Centre for Human Brain Health, University of Birmingham, Birmingham, UK. · School of Psychology, University of Birmingham, Birmingham, UK. · Institute for Mental Health, University of Birmingham, Birmingham, UK. +3

Social ReasoningTheory of Mind
AskPDF
Apr 30, 2026

Why Do LLMs Struggle in Strategic Play? Broken Links Between Observations, Beliefs, and Actions

Large language models (LLMs) are increasingly tasked with strategic decision-making under incomplete information, such as in negotiation and policymaking. While LLMs can excel at many such tasks, they also fail in ways that are poorly understood. We shed light on these failures by uncovering two fundamental gaps in the internal mechanisms underlying the decision-making of LLMs in incomplete-information games, supported by experiments with open-weight models Llama 3.1, Qwen3, and gpt-oss. First, an observation-belief gap: LLMs encode internal beliefs about latent game states that are substantially more accurate than their own verbal reports, yet these beliefs are brittle. In particular, the belief accuracy degrades with multi-hop reasoning, exhibits primacy and recency biases, and drifts away from Bayesian coherence over extended interactions. Second, a belief-action gap: The implicit conversion of internal beliefs into actions is weaker than that of the beliefs externalized in the prompt, yet neither belief-conditioning consistently achieves higher game payoffs. These results show how analyzing LLMs' internal processes can expose systematic vulnerabilities that warrant caution before deploying LLMs in strategic domains without robust guardrails.

Jan Sobotka, Mustafa O. Karabag, Ufuk Topcu

EPFL · The University of Texas at Austin

Imperfect-Information GamesLLM InterpretabilityLLM Decision-Making
AskPDF
Apr 13, 2026

Playing Along: Learning a Double-Agent Defender for Belief Steering via Theory of Mind

As large language models (LLMs) become the engine behind conversational systems, their ability to reason about the intentions and states of their dialogue partners (i.e., form and use a theory-of-mind, or ToM) becomes increasingly critical for safe interaction with potentially adversarial partners. We propose a novel privacy-themed ToM challenge, ToM for Steering Beliefs (ToM-SB), in which a defender must act as a Double Agent to steer the beliefs of an attacker with partial prior knowledge within a shared universe. To succeed on ToM-SB, the defender must engage with and form a ToM of the attacker, with a goal of fooling the attacker into believing they have succeeded in extracting sensitive information. We find that strong frontier models like Gemini3-Pro and GPT-5.4 struggle on ToM-SB, often failing to fool attackers in hard scenarios with partial attacker prior knowledge, even when prompted to reason about the attacker's beliefs (ToM prompting). To close this gap, we train models on ToM-SB to act as AI Double Agents using reinforcement learning, testing both fooling and ToM rewards. Notably, we find a bidirectionally emergent relationship between ToM and attacker-fooling: rewarding fooling success alone improves ToM, and rewarding ToM alone improves fooling. Across four attackers with different strengths, six defender methods, and both in-distribution and out-of-distribution (OOD) evaluation, we find that gains in ToM and attacker-fooling are well-correlated, highlighting belief modeling as a key driver of success on ToM-SB. AI Double Agents that combine both ToM and fooling rewards yield the strongest fooling and ToM performance, outperforming Gemini3-Pro and GPT-5.4 with ToM prompting on hard scenarios. We also show that ToM-SB and AI Double Agents can be extended to stronger attackers, demonstrating generalization to OOD settings and the upgradability of our task.

Hanqi Xiao, Vaidehi Patil, Zaid Khan +3

UNC Chapel Hill · University of Texas at Austin

Theory of MindAI Agent SafetyPrivacy-Preserving Language Models
AskPDF