Playing Along: Learning a Double-Agent Defender for Belief Steering via Theory of Mind

Hanqi Xiao; Vaidehi Patil; Zaid Khan; Hyunji Lee; Elias Stengel-Eskin; Mohit Bansal

cs.CLApr 13, 2026

Playing Along: Learning a Double-Agent Defender for Belief Steering via Theory of Mind

Authors: Hanqi Xiao, Vaidehi Patil, Zaid Khan, Hyunji Lee, Elias Stengel-Eskin, Mohit Bansal

Organizations: UNC Chapel Hill · University of Texas at Austin

Abstract

As large language models (LLMs) become the engine behind conversational systems, their ability to reason about the intentions and states of their dialogue partners (i.e., form and use a theory-of-mind, or ToM) becomes increasingly critical for safe interaction with potentially adversarial partners. We propose a novel privacy-themed ToM challenge, ToM for Steering Beliefs (ToM-SB), in which a defender must act as a Double Agent to steer the beliefs of an attacker with partial prior knowledge within a shared universe. To succeed on ToM-SB, the defender must engage with and form a ToM of the attacker, with a goal of fooling the attacker into believing they have succeeded in extracting sensitive information. We find that strong frontier models like Gemini3-Pro and GPT-5.4 struggle on ToM-SB, often failing to fool attackers in hard scenarios with partial attacker prior knowledge, even when prompted to reason about the attacker's beliefs (ToM prompting). To close this gap, we train models on ToM-SB to act as AI Double Agents using reinforcement learning, testing both fooling and ToM rewards. Notably, we find a bidirectionally emergent relationship between ToM and attacker-fooling: rewarding fooling success alone improves ToM, and rewarding ToM alone improves fooling. Across four attackers with different strengths, six defender methods, and both in-distribution and out-of-distribution (OOD) evaluation, we find that gains in ToM and attacker-fooling are well-correlated, highlighting belief modeling as a key driver of success on ToM-SB. AI Double Agents that combine both ToM and fooling rewards yield the strongest fooling and ToM performance, outperforming Gemini3-Pro and GPT-5.4 with ToM prompting on hard scenarios. We also show that ToM-SB and AI Double Agents can be extended to stronger attackers, demonstrating generalization to OOD settings and the upgradability of our task.

Figures & tables

Figure 1: Illustration of our multi-layer scenario and attacker-defender interaction dynamics. The attacker begins with partial prior knowledge (e.g., division) and incrementally probes for finer-grained information (e.g., department, team). Defenders must protect private fine-grained information (top-left) that the attacker does not know. Reactive defenses (a) refuse to answer and is detected as a defender by the attacker, while naive proactive defenses (b) introduce inconsistencies that reduce trust and trigger detection. In contrast, the AI Double Agents defender (c) performs belief-consistent deception, aligning with the attacker’s prior beliefs while steering them toward a consistent but incorrect inference.

Figure 2: Illustration of metrics and reward computation. For each trajectory, the ToM accuracy reward (trajectory-wise) evaluates how well the defender’s estimate of the attacker’s prior belief matches the attacker’s actual prior belief. The fooling reward evaluates whether the attacker has been fooled. The Stepwise-ToM accuracy metric computes the alignment between the defender’s estimate of the attacker’s belief at each turn and the attacker’s actual beliefs at that turn, and is averaged across turns (note: universe is shared with Figure 1 ).

Method	Fooling %		ToM Acc %		Avg Turns
	Hard $\uparrow$	All $\uparrow$	Traj $\uparrow$	Step $\uparrow$
Prompt-based
Base Prompt (Gemini3-Pro)	$34.4\,{\scriptstyle\pm\,3.1}$	$57.8\,{\scriptstyle\pm\,2.3}$	$48.9\,{\scriptstyle\pm\,2.3}$	$47.0\,{\scriptstyle\pm\,1.5}$	$3.60\,{\scriptstyle\pm\,0.13}$
Base Prompt (GPT-5.4)	$26.7\,{\scriptstyle\pm\,1.3}$	$51.1\,{\scriptstyle\pm\,0.6}$	$49.8\,{\scriptstyle\pm\,2.5}$	$49.9\,{\scriptstyle\pm\,0.9}$	$3.53\,{\scriptstyle\pm\,0.04}$
Refuse Prompt	$0.3\,{\scriptstyle\pm\,0.5}$	$0.2\,{\scriptstyle\pm\,0.3}$	–	–	$5.94\,{\scriptstyle\pm\,0.09}$
Mislead Prompt	$4.2\,{\scriptstyle\pm\,1.5}$	$37.3\,{\scriptstyle\pm\,1.9}$	–	–	$2.69\,{\scriptstyle\pm\,0.06}$

Table 1: Qwen3-14B defender vs baseline Gemini3-flash attacker. For fooling %, “Hard” refers to evaluation only on the hard scenarios, where the attacker has at least 1 piece of prior knowledge. ToM Acc % reports ToM trajectory-wise (Traj) and ToM stepwise (Step). All metrics on Qwen3-14B (

\mathrm{mean}\pm\mathrm{std}

) averaged across three training seeds. All ToM Acc % and fooling % metrics are reported as percentages. Mislead and Refuse prompts do not output ToM reflections and thus have no ToM accuracy scores.

Prompt-based
Method	Base		Cross-Examiner		Deception-Aware		Bluffing
	Fooling $\uparrow$	ToM $\uparrow$	Fooling $\uparrow$	ToM $\uparrow$	Fooling $\uparrow$	ToM $\uparrow$	Fooling $\uparrow$	ToM $\uparrow$
Base Prompt (Gemini3-Pro)	$34.4$	$48.9$	35.4	$54.0$	$24.0$	$46.0$	$20.8$	$24.7$
Mislead Prompt	$4.2$	–	$3.1$	–	$6.2$	–	$6.2$	–
Base Prompt	$13.2$	$36.0$	$15.6$	$40.7$	$16.7$	$40.7$	$19.8$	$22.7$
Training-based

Table 2: Fooling rate and ToM accuracy by attacker. Fooling rate is evaluated on the hard scenarios, where the attacker has at least 1 piece of prior knowledge. ToM denotes trajectory-level ToM accuracy. All training-based rows are AI Double Agents variants trained on the Base attacker. All metrics are reported as percentages.

Figure 3: Fooling ability of a defender is positively correlated with its ToM accuracy. The plot shows fooling % (hard) vs ToM accuracy % (trajectory-wise) across four attacker variants.

Method	Fooling %		ToM Acc %		Avg Turns
	Hard $\uparrow$	All $\uparrow$	Traj $\uparrow$	Step $\uparrow$
Base Prompt ( Gemini3-Pro )	$20.8$	$28.0$	$24.7$	$45.6$	$5.27$
Base Prompt	$19.8$	$34.0$	$22.7$	$59.2$	$4.57$
ADA (Fooling + ToM)	34.4	34.7	25.3	62.4	$4.79$

Table 3: We evaluate an ADA trained against the Bluffing Attacker, illustrating that training also becomes more difficult as attacker capability increases (Compare Table 1 ).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Fooling %		ToM Acc %		Avg Turns
	Hard $\uparrow$	All $\uparrow$	Traj $\uparrow$	Step $\uparrow$
GPT-5.4 vs Base Attacker	$26.7\,{}$	$51.1\,{}$	$49.8\,{}$	$49.9\,{}$	$3.53$
GPT-5.4 vs Cross-Examiner	$16.7$	$45.3$	$44.7$	$49.9$	$4.20$
GPT-5.4 vs Deception-Aware	$6.2$	$38.7$	$40.7$	$46.2$	$4.15$
GPT-5.4 vs Bluffing Attacker	$15.6$	$24.7$	$22.0$	$45.6$	$4.99$

Appendix

Table 4: GPT-5.4 defender vs all types of attackers.

Method	Fooling %		ToM Acc %		Avg Turns
	Hard $\uparrow$	All $\uparrow$	Traj $\uparrow$	Step $\uparrow$
Base Prompt	$11.5\,{}$	$36.7\,{}$	$42.0\,{}$	$46.5\,{}$	$3.09$
ADA - Fooling	$57.3\,{}$	$69.3\,{}$	$60.7\,{}$	$60.4\,{}$	$4.19$
ADA - ToM Only	$62.5\,{}$	$72.7\,{}$	$72.0\,{}$	$66.0\,{}$	$3.89\,{}$

Appendix

Table 5: Ministral-3-14B-Instruct based ADA Defenders vs base Gemini3-Flash attacker. We show bi-directional emergence described in Section 6.1 holds also for Ministral-3-14B-Instruct models.

Method	Fooling % (Hard)	95% CI for Improvement	Fooling % (All)	95% CI for Improvement
Base Prompt	$16.7$	n/a	$37.3$	n/a
Base Prompt + Oracle ToM	$\bf{59.4}$	$27.1$ to $57.3$	$\bf{68.0}$	$18.7$ to $42.7$

Appendix

Table 6: Effect of injecting the ground-truth attacker prior knowledge (oracle ToM) into the Qwen3-14B Base Prompt defender’s prompt at inference time, evaluated against the base attacker. We report bootstrap 95% confidence intervals over evaluation scenarios for the improvement over the Base Prompt defender. Results are from a single evaluation run under conditions identical to the intervention setting.

Metric	Comparison (A $>$ B)	Improvement per seed	Mean improvement	One-sided paired t-test $p$	95% CI for improvement
Fooling (All)	Fooling+ToM $>$ Fooling only	$+7.33\%$ , $+4.67\%$ , $+2.00\%$	$+4.67\%$	$\bf{0.047}$	$-1.96\%$ to $11.29\%$
Fooling (All)	Fooling+ToM $>$ ToM only	$+12.00\%$ , $+6.00\%$ , $+5.33\%$	$+7.78\%$	$\bf{0.033}$	$-1.34\%$ to $16.90\%$
ToM (Traj)	Fooling+ToM $>$ Fooling only	$+9.33\%$ , $+6.90\%$ , $+12.66\%$	$+9.63\%$	$\bf{0.014}$	$2.45\%$ to $16.81\%$
ToM (Traj)	Fooling+ToM $>$ ToM only	$+3.33\%$ , $+2.66\%$ , $+10.00\%$	$+5.33\%$	$0.075$	$-4.75\%$ to $15.41\%$

Appendix

Table 7: Paired seed-level comparisons of ADA (Fooling + ToM) against single-reward training across the three training seeds in Table 1 . We report the improvement for each seed, the mean improvement, one-sided paired t-test

p

-values (bolded when

p<0.05

), and two-sided 95% confidence intervals for the mean improvement.

Method	Fooling % (All)	ToM Acc % (Traj)
Base Prompt (no strategic hints)	$28.0$	$26.0$
ADA - ToM Only (no strategic hints)	$34.7$	$29.3$
ADA - Fooling (no strategic hints)	$42.7$	$39.3$

Appendix

Table 8: Qwen3-14B defenders prompted and trained with all strategic hints removed from the defender prompt, evaluated against the base attacker. Training with either reward alone improves both fooling and ToM, reproducing the bidirectional emergence observed in Table 1 .

Attacker Base Model	Fooling % (All)	ToM Acc % (Traj)
Gemini3-Flash (base)	$36.0$	$36.0$
Gemini-3.5-Flash	$38.7$	$34.7$
Gemini-3.1-Pro	$42.0$	$39.3$
GPT-5.4	$50.7$	$40.7$

Appendix

Table 9: Qwen3-14B Base Prompt defender evaluated against attackers powered by different base models under the same base attacker prompt. Stronger attacker base models do not make the attacker more difficult to fool.

Defender Model	Fooling % (No Shared Universe)
Gemini3-Pro	$100.0$
Gemini-2.5-Flash	$100.0$
Qwen3-32B	$85.7$

Appendix

Table 10: Fooling rates of prompted defenders in an early version of the task without a shared universe. Without a shared universe, defenders can fabricate unverifiable lies and fooling becomes degenerate.

Method	Fooling %		ToM Acc %		Avg Turns
	Hard $\uparrow$	All $\uparrow$	Traj $\uparrow$	Step $\uparrow$
Prompt-based
Base Prompt ( Gemini3-Pro )	$34.4$	$57.8$	$48.9$	$47.0$	$3.60$
Mislead Prompt	$4.2$	$37.3$	–	–	$2.69$
Base Prompt	$13.2$	$36.0$	$36.0$	$49.3$	$3.12$
Training-based

Appendix

Table 11: Base Attacker. Qwen3-14B Defender vs. Gemini3-Flash attacker.

Method	Fooling %		ToM Acc %		Avg Turns
	Hard $\uparrow$	All $\uparrow$	Traj $\uparrow$	Step $\uparrow$
Prompt-based
Base Prompt ( Gemini3-Pro )	$35.4$	$58.0$	$54.0$	$52.5$	$4.07$
Mislead Prompt	$3.1$	$34.0$	–	–	$3.71$
Base Prompt	$15.6$	$35.3$	$40.7$	$52.6$	$4.11$
Training-based

Appendix

Table 12: Cross-Examiner. Qwen3-14B Defender vs. Gemini3-Flash attacker.

Method	Fooling %		ToM Acc %		Avg Turns
	Hard $\uparrow$	All $\uparrow$	Traj $\uparrow$	Step $\uparrow$
Prompt-based
Base Prompt ( Gemini3-Pro )	$24.0$	$48.7$	$46.0$	$54.9$	$5.09$
Mislead Prompt	$6.2$	$37.3$	–	–	$3.49$
Base Prompt	$16.7$	$39.3$	$40.7$	$59.7$	$3.85$
Training-based

Appendix

Table 13: Deception-Aware Attacker. Qwen3-14B Defender vs. Gemini3-Flash attacker.

Method	Fooling %		ToM Acc %		Avg Turns
	Hard $\uparrow$	All $\uparrow$	Traj $\uparrow$	Step $\uparrow$
Prompt-based
Base Prompt ( Gemini3-Pro )	$20.8$	$28.0$	$24.7$	$45.6$	$5.27$
Mislead Prompt	$6.2$	$26.7$	–	–	$3.65$
Base Prompt	$19.8$	$34.0$	$22.7$	$59.2$	$4.57$
Training-based

Appendix

Table 14: Bluffing Attacker. Qwen3-14B Defender vs. Gemini3-Flash attacker.

Figure 4: Replication of Fig. 3 for all pairings of ToM and fooling metrics. We consider four metrics in total in the main paper, leading to six pairs of correlation plots. The setting and datapoints used are the same as in Fig. 3 . Values are sourced from Table 11 to Table 14 .

Theory of Mind

Theory of Mind studies how people and AI infer others’ beliefs, goals, and knowledge to predict behavior and coordinate interaction.

See all 92 papers

May 19, 2026

OSCToM: RL-Guided Adversarial Generation for High-Order Theory of Mind

Large Language Models (LLMs) perform well on many language tasks, but their Theory of Mind (ToM) reasoning is still uneven in complex social settings. Existing benchmarks, including ExploreToM, do not always test the recursive beliefs and information asymmetries that make these settings difficult. This paper presents OSCToM (Observer-Self Conflict Theory of Mind), an approach for modeling nested belief conflicts in LLM-based ToM tasks. The key case is one in which an observer's view of another agent conflicts with the observer's own belief state. Such cases go beyond simple perspective-taking and require recursive, multi-layered reasoning. OSCToM combines reinforcement learning (RL), an extended domain-specific language, and compositional surrogate models to generate observer-self conflicts. In our experiments, OSCToM-8B gives the best overall result among the systems tested. It improves on the reported ExploreToM results on FANToM and remains competitive on Hi-ToM and BigToM. On the information-asymmetric FANToM benchmark, OSCToM reaches 76% accuracy, compared with the 0.2% reported by ExploreToM. The data-synthesis procedure is also 6x more efficient, indicating that targeted training data can help smaller models handle advanced cognitive reasoning. The project code is available at https://github.com/sharminsrishty/osct.

Sharmin Sultana Srishty, Kazi Mahathir Rahman, Malaika Parizat Sakkhi +2

Department of Computer Science BRAC University

Cognitive Modeling Theory of Mind RL for Language Model Reasoning

Ask PDF

Jun 30, 2026

Theory of Mind and Persuasion Beyond Conversation: Assessing the Capacity of LLMs to Induce Belief States via Planning and Action

Theory of Mind (ToM) benchmarks for Large Language Models (LLMs) typically rely on passive question-answering formats, but the deployment of LLMs in increasingly agentic and autonomous forms demands new evaluations. In this paper we evaluate an agent's ability to induce specific belief states in other agents by taking actions rather than using conversational persuasion, a capability we call Non-Conversational Planning ToM (NCP-ToM). NCP-ToM is likely to be essential for many agent use-cases, including within user-assistant interactions and pedagogical contexts, but may also present manipulation or misinformation risks. Using a novel framework, NCP-ExploreToM, we subvert the conventional task structure by providing models with a set of belief state goals and requiring them to move objects or direct characters into rooms to achieve their goals. We evaluated six frontier models, including GPT-5, Gemini 2.5 Pro and the Claude 4 series, and a cohort of human participants, across 600 task instances. GPT-5 was successful on approximately 80% of tasks in the agentic setting, and was the only model to outperform human participants on our task, but was still less robust than humans across contexts. We additionally found that all models, like humans, performed better on tasks inducing true belief states than false belief states, which is a positive signal for alignment efforts. These findings highlight emerging social-reasoning capabilities in LLMs for non-conversational task completion and underscore the necessity of agentic evaluations for understanding the safety and alignment of autonomous social agents.

Ben Slater, Matteo G. Mecattaf, Lucy G. Cheke +2

Leverhulme Centre for the Future of Intelligence Department of Psychology University of Cambridge, UK · Prolific · Google, Paradigms of Intelligence Team

Theory of Mind LLM Agent Evaluation Persuasion in Language Models LLM Agent Planning

Ask PDF

Feb 25, 2026

Understanding Artificial Theory of Mind: Perturbed Tasks and Reasoning in Large Language Models

Theory of Mind (ToM) refers to an agent's ability to model the internal states of others. Contributing to the debate whether large language models (LLMs) exhibit genuine ToM capabilities, our study investigates their ToM robustness using perturbations on false-belief tasks and examines the potential of Chain-of-Thought prompting (CoT) to enhance performance and explain the LLM's decision. We introduce a handcrafted, richly annotated ToM dataset, including classic and perturbed false belief tasks, the corresponding spaces of valid reasoning chains for correct task completion, subsequent reasoning faithfulness, task solutions, and propose metrics to evaluate reasoning chain correctness and to what extent final answers are faithful to reasoning traces of the generated CoT. We show a steep drop in ToM capabilities under task perturbation for all evaluated LLMs, questioning the notion of any robust form of ToM being present. While CoT prompting improves the ToM performance overall in a faithful manner, it surprisingly degrades accuracy for some perturbation classes, indicating that selective application is necessary.

Christian Nickel, Laura Schrewe, Florian Mai +1

Bonn-Aachen International Center for Information Technology (b-it), University of Bonn · Lamarr Institute for Machine Learning and Artificial Intelligence · Research Center Trustworthy Data Science and Security (RC-Trust), University of Duisburg-Essen

Theory of Mind Language Model Robustness Faithfulness of Language Model Explanations CoT Reasoning

Ask PDF

Abstract

Theory of Mind

Explore similar work

OSCToM: RL-Guided Adversarial Generation for High-Order Theory of Mind

Theory of Mind and Persuasion Beyond Conversation: Assessing the Capacity of LLMs to Induce Belief States via Planning and Action

Understanding Artificial Theory of Mind: Perturbed Tasks and Reasoning in Large Language Models