Forming an accurate theory-of-mind (ToM) is key to successful multi-agent collaboration and requires reasoning under uncertainty to capture multiple possible partner states. This is hard to do in the text representation space native to large language model (LLM) agents, which more easily describe a single partner state. To address this challenge, we introduce ToM2Vec, which learns to form abstract theories-of-mind in a rich latent space via a fixed budget of continuous reasoning tokens. During training, supervision for these tokens comes from the partner's actual thoughts, which the ToM tokens are trained to abstract and match. At test-time, ToM tokens are generated by each agent and act as conditioners for that agent's thoughts and actions. Across several environments covering collaborative, competitive, and mixed-motive settings (Overcooked-AI, multiparty negotiation, Decrypto, Hanabi, and Collab Escape/Capture), we evaluate ToM2Vec's ability to support multi-agent interactions and collaboration, finding consistent improvements over the base model and a variety of competitive trained and prompted baselines. We show that latent reasoning is critical, with performance dropping when reasoning is expressed in text. Moreover, we show that ToM2Vec training generalizes to partner model families not seen at train-time, including a proprietary model (GPT-5.4). In our analysis, we provide evidence that the latent tokens encode a ToM, rather than task-specific information: ToM2Vec's tokens improve the agent's ability to directly answer questions about partner beliefs, desires, and next actions.
Figures & tables
Figure 1: Left : Vanilla LLM agents (using Qwen3-8B) each reason only about their own plan. Both independently conclude they should fetch a dish from the dispenser, producing redundant and conflicting behavior. Middle : Language ToM Agents are able to more frequently consider their partner’s next actions, but they often commit to a single line of reasoning without considering other potential paths. Right : Our method ToM2Vec is used to train an agent (blue hat) with stronger ToM capabilities; that agent considers what the untrained partner agent (green hat) is doing before making its own decision.
Figure 2: Overview of ToM2Vec on the multi-party negotiation task. Left (training): From its local observation and interaction history, the main agent predicts K continuous latent ToM tokens that summarize a partner’s likely reasoning trace. Middle (training): The partner’s reasoning is segmented at sentence delimiters, and latent tokens are extracted across the trace. These K vectors provide self-supervised targets for position-wise cosine alignment with the main agent’s predictions. Right (inference): The main agent predicts the latent ToM tokens using only its local context and conditions its subsequent reasoning and action on them. The partner’s private context and reasoning are used only to construct training targets.
Model
CrR
AA
CoR
Avg.
Untrained
90.0±1.86
91.3±3.28
102.7±1.26
94.7
Prompted ToM
86.7±1.75
96.0±3.64
106.7±1.75
96.4
Language ToM
92.0±2.81
98.7±0.93
102.0±1.11
97.6
Self-Consistency
91.3±3.28
97.3±1.26
107.3±1.79
98.7
SimToM
90.7±1.85
94.0±1.70
105.3±1.64
96.7
Adaptive ToM
92.7±1.79
102.7±0.93
101.3±2.34
98.9
Table 1: Overcooked mean episode score and standard error on three layouts and their average score across 30 seeds. All settings use Qwen3-8B. We bold the best mean episode score for each layout. CrR, AA, and CoR stand for the Cramped Room, Asymmetric Advantages, and Coordination Ring layouts respectively. † Oracle has privileged access to the partner’s context and serves as an upper bound.
Method
Deal %
Own Score
Collect. Score
Untrained
62.0
59.2
54.7
Prompted ToM
68.0
61.5
56.8
Language ToM
64.0
60.8
59.2
Self-Consistency
63.0
58.4
58.0
SimToM
65.0
62.1
58.4
ToM2Vec
71.0
65.6
59.2
Table 2: Evaluation results for multi-party, multi-issue negotiation ( Abdelnabi et al., 2024 ) across 100 negotiations. Deal % is the fraction of games ending in an agreement (at least 5/6 agents must agree). Own and Collective Score are the agent’s own average final score and the average per-party utility from the final deal.
Model
Decoder Win %
Interceptor Win %
Untrained
10.4%
87.4%
Prompted ToM
12.8%
88.2%
Language ToM
13.6%
93.0%
Self-Consistency
12.0%
90.5%
SimToM
11.5%
92.6%
ToM2Vec
14.8%
93.5%
Table 3: Decrypto results: win rates for each method as the Decoder and Interceptor for 1000 samples each. In all settings, the other agents are Qwen3-8B.
Model
Score
Untrained
8.9
Prompted ToM
9.6
Language ToM
10.6
Self-Consistency
9.5
SimToM
10.3
ToM2Vec
11.3
Table 4: Hanabi scores (out of 25). Higher is better, highest score bolded.
Collab Capture
Collab Escape
Model
Capture Rate
Avg. Turns
Escape Rate
Avg. Turns
Untrained
0.67
5.00
0.75
8.00
Prompted ToM
1.00
4.76
0.75
8.74
Language ToM
0.67
5.52
0.67
8.24
Self-Consistency
0.83
5.21
0.75
8.00
SimToM
0.83
5.00
0.92
7.80
Table 5: Results on Collab Capture and Collab Escape. Success rates are reported as fractions. We bold the highest success rate and lowest average turn count in each game, including ties; fewer turns is better.
Model
Partner
CrR
AA
CoR
Qwen3-8B
Qwen3-8B
89
93
105
ToM2Vec
98
108
113
GPT-5.4
Qwen3-8B
135
193
174
ToM2Vec
152
214
188
Gemma-4
Qwen3-8B
65
90
87
ToM2Vec
68
105
94
Table 6: Overcooked results for partner agents of different architectures across 20 seeds. Each agent is tested with both Qwen3-8B and ToM2Vec . Neither GPT-5.4 nor Gemma-4 were seen during training.
Model
Beliefs (%)
Desires (%)
Actions (%)
Random
16.7
16.7
16.7
Untrained
41.7
53.0
55.3
Prompted ToM
40.0
55.0
56.3
Language ToM
46.7
59.3
61.3
Self-Consistency
44.0
54.7
57.7
SimToM
43.7
54.7
58.0
Table 7: Accuracy on Bayesian ToM probes in Overcooked. Beliefs : what the partner believes the agent is trying to do (a second-order inference). Desires : which subtask the partner is currently pursuing. Actions : which primitive action the partner will take next.
Thought Type
PPL ( ↓ )
Lowest PPL Rate
Normal
1.412
98.6%
Altered Layout
1.702
0.6%
Adversarial Intent
1.613
0.8%
Table 8: Perplexity of generated thoughts when trained by ToM2Vec across different types of thoughts. We test on the normal thoughts, altered layout (swapped real layout for alternate layout), and adversarial intent (partner agent is attempting to sabotage).
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Accuracy
State Understanding
Untrained
0.74
0.88
Prompted ToM
0.82
0.86
Language ToM
0.86
0.90
Self-Consistency
0.88
0.94
SimToM
0.84
0.88
Adaptive ToM
0.80
0.92
Appendix
Table 9: Results on the Sally-Anne belief test for different baselines. We present the accuracy as the proportion of output responses that the overseer agents correctly recognized where the first player would look. We also show the state understanding to illustrate how well each method keeps track of each object.
Figure 3: We design a custom layout and collect five east/west CoTs where the partner (green hat) decides to go either east towards the dish dispenser or west towards the onion dispenser. We calculate the perplexity of each CoT, plotted below. A high east perplexity indicates that a CoT corresponding to the dish dispenser is unlikely ; a low east perplexity means a dish CoT is likely, and vice versa for west and onions.
Baseline
CrR
AA
CoR
Untrained
0.008
0.000
0.000
Prompted ToM
0.000
0.006
0.013
Language ToM
0.048
0.000
0.000
Self-Consistency
0.115
0.000
0.026
SimToM
0.032
0.000
0.001
Adaptive ToM
0.088
0.001
0.000
Appendix
Table 10: Paired t -test p -values comparing ToM2Vec against each baseline over 30 matched random seeds. CrR, AA, and CoR denote Cramped Room, Asymmetric Advantages, and Coordination Ring, respectively. Values are rounded to three decimal places.
Baseline
p -value
Untrained
0.078
Prompted ToM
0.607
Language ToM
0.143
Self-Consistency
0.115
SimToM
0.180
Appendix
Table 11: McNemar test p -values over 100 samples comparing ToM2Vec to each baseline for deal success in the multi-party multi-issue negotiation task.
Baseline
Decoder
Interceptor
Untrained
0.000
0.000
Prompted ToM
0.046
0.000
Language ToM
0.180
0.500
Self-Consistency
0.005
0.000
SimToM
0.000
0.285
Appendix
Table 12: McNemar test p -values over 1000 samples comparing ToM2Vec to each baseline for both Decoder and Interceptor roles in Decrypto.
Baseline
p -value
Untrained
0.004
Prompted ToM
0.018
Language ToM
0.216
Self-Consistency
0.011
SimToM
0.079
Appendix
Table 13: Paired t -test p -values comparing ToM2Vec to each baseline in Hanabi over 100 seeds.
Metric
Qwen3-8B
ToM2Vec
Per-Query Latency (ms)
11491.8
12641.9
Inference peak memory (GB)
16.3
16.3
Training peak memory (GB)
N/A
84.0
Appendix
Table 14: Per-Query Latency, inference memory, and training peak memory (over 50 steps) for untrained Qwen3-8B and ToM2Vec .
Theory of Mind (ToM) refers to the ability to reason about others' mental states, and higher-order ToM involves considering that others also possess their own ToM. Equipping large language model (LLM)-driven agents with ToM has long been considered to improve their coordination in multiagent collaborative tasks. However, we find that misaligned ToM orders-mismatches in the depth of ToM reasoning between agents-can lead to insufficient or excessive reasoning about others, thereby impairing their coordination. To address this issue, we design an adaptive ToM (A-ToM) agent, which can align in ToM orders with its partner. Based on prior interactions, the agent estimates the partner's likely ToM order and leverages this estimation to predict the partner's action, thereby facilitating behavioral coordination. We conduct empirical evaluations on four multi-agent coordination tasks: a repeated matrix game, two grid navigation tasks and an Overcooked task. The results validate our findings on ToM alignment and demonstrate the effectiveness of our A-ToM agent. Furthermore, we discuss the generalizability of our A-ToM to non-LLM-based agents, as well as what would diminish the importance of ToM alignment.
Chunjiang Mu, Ya Zeng, Qiaosheng Zhang +6
School of Cybersecurity, Northwestern Polytechnical University · Shanghai Artificial Intelligence Laboratory · Huawei Noah’s Ark Lab +2
Theory of Mind (ToM), the ability to track others epistemic state, makes humans efficient collaborators. AI agents need the same capacity in multi agent settings, yet existing benchmarks mostly test literal ToM by asking direct belief questions. The ability act optimally on implicit beliefs in embodied environments, called functional ToM, remains largely untested. We introduce EnactToM, an evolving benchmark of 300 embodied multi-agent tasks set in a 3D household with partial observability, private information, and constrained communication. Each task is formally verified for solvability and required epistemic depth, and new tasks are generated increase difficulty as models improve. On the hard split, all seven evaluated frontier models score 0.0% Pass^3 on functional task completion, while averaging 45.0% on literal belief probes. Manual analysis traces 93% of sampled failures to epistemic coordination breakdowns such as withheld information, ignored partner constraints, and misallocated messages, providing a concrete target for future work.
Gurusha Juneja, Dylan Lu, Saaket Agashe +7
University of California, Santa Barbara · 2King’s College London · 3Cisco Research
Large foundation models have accelerated progress toward general-purpose agents that interact with humans and other agents through language and multimodal signals. However, robust multi-agent decision-making requires reasoning about what other agents know, intend, and are likely to do under partial observability. Current agentic systems often operate through prompt design, memory, or end-to-end behavioral shaping, but typically do not learn an explicit partner-state representation that can be reused as a decision variable across tasks. We introduce mental-model-enabled agents, a framework that equips an agent with a latent mental model of its counterpart, allowing it to infer hidden beliefs, intentions, and likely reactions from the observed history and use these inferences to guide action selection. Our method learns an amortized recursive Theory-of-Mind representation, with first- and second-order mental-state structure, jointly with a belief-conditioned reward model that evaluates candidate actions relative to the inferred partner state. A policy is then learned under this belief-aware signal, yielding an agent that can act independently at inference time while retaining the benefits of explicit partner modeling. We evaluate the same framework on both language-only and multimodal benchmarks. Across these settings, explicit mental-state modeling consistently improves interaction quality and Theory-of-Mind performance over base agentic systems, showing that structured partner modeling is a useful inductive bias for general multi-agent systems. Our code is publicly available at https://github.com/hananshafi/Mental-Models