Belief Without Behavior: Measuring the Translation of Theory of Mind into Coordinated Social Action in Vision-Language Models

Tonglin Yan; Gregoire Sergeant-Perthuis; David Rudrauf

cs.AIAug 21, 2026

Belief Without Behavior: Measuring the Translation of Theory of Mind into Coordinated Social Action in Vision-Language Models

Authors: Tonglin Yan, Gregoire Sergeant-Perthuis, David Rudrauf

Organizations: CIAMS, Université Paris-Saclay · CQSB, Sorbonne Université

Abstract

Effective social interaction requires agents to translate mental state inferences into coordinated behavioral signals across verbal and nonverbal channels simultaneously. Yet existing benchmarks evaluate theory of mind (ToM) reasoning and embodied behavior in isolation, leaving unmeasured the gap between social inference and social action. We introduce MOSAIC (Multimodal Orchestration of Social Action, Inference, and Communication), a controlled benchmark in which two embodied agents interact across cooperative and competitive scenarios requiring integration of verbal statements, spatial trajectories, gaze direction, and facial expression under systematically varied ToM constraints. Evaluating 13 models, including 11 VLMs, across 200 trials per model, we find that VLMs fail to produce behaviors consistent with the expected outcomes under ToM-order constraints, and that imposing explicit ToM-order constraints produces no reliable behavioral change aligned with the specified reasoning level. Signal-level analysis reveals two sequential bottlenecks: most models cannot produce directionally coherent nonverbal signals, and even when signals are present, VLM agents fail to interpret others behaviors and react to them. PCM-LLM, included as a structured architectural reference point with an explicit ToM module, succeeds across all conditions, suggesting that explicit belief-action coupling is a sufficient ingredient for this class of tasks.

Figures & tables

Figure 1: MOSAIC benchmark design. (Top left) Two agents interact in a virtual environment: the subject knows the reward location and must help or mislead the participant depending on the assigned relationship mode. (Bottom left) Four experimental conditions cross interaction mode (competitive vs. cooperative) with ToM constraint level (ToM-0 vs. ToM-1), with 50 trials per condition per model. (Right) Each trial runs for 10 rounds; rounds 3, 6, and 9 include a verbal exchange phase besides non-verbal actions. In non-verbal rounds, each agent receives a context prompt, a first-person perspective visual perception, and a structured belief-and-observation state, and produces action reasoning together with motor action (movement, rotation, or staying idle) and emotional expression (musculoskeletal and physiological channels). In verbal rounds, the same inputs are supplemented by the interlocutor’s speech, and the agent generates an inner speech reasoning trace and a natural language answer.

Figure 2: ToM Outcome Conformance Score (TOCS) with positional bias reported. Positional bias is measured as the proportion of non-neutral trials in which the model selects its modal box. High TOCS with high positional bias indicates that alignment may be attributable to a fixed spatial preference rather than signal-responsive decision-making; high TOCS with low positional bias suggests genuine condition-sensitive behavior.

Figure 3: Signal-level metrics across models and experimental conditions. Panels (a) and (b) show Trajectory Alignment Score (TAS) and Gaze Alignment Score (GAS), where positive values indicate signals directed toward the reward box, negative values indicate misdirection, and values near zero reflect ambiguous or absent directional content. Panel (c) shows Facial Expressivity Score (FES), where higher values reflect greater affective activity regardless of spatial direction. Panel (d) shows Signal Sensitivity Score (SSS), where positive values indicate that the participant’s final choice is consistent with the subject’s nonverbal signals, values near zero reflect chance-level correspondence, and negative values indicate systematic opposition.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: The PCM-LLM framework consists of three modules: Unity, PCM, and an LLM. PCM runs at Unity’s tick rate, receives sensory evidence each iteration, and selects the agent’s next action. LLMs are engaged for verbal interactions. The PCM-LLM interface performs structured, bidirectional information exchange between the nonverbal PCM representation and the LLM.

Benchmark	Type	MT	LH	Emb	MM	S-ToM	CV	Aff
Hi-ToM 1	ToM						$\triangle$
FANToM 2	ToM	$\checkmark$
MMToM-QA 3	ToM			$\triangle$	$\checkmark$			$\triangle$
MuMA-ToM 4	ToM			$\triangle$	$\checkmark$			$\triangle$
SOTOPIA 5	MA	$\checkmark$	$\triangle$			$\triangle$
MindCraft 6	MA	$\checkmark$	$\triangle$	$\triangle$		$\triangle$

Appendix

Table 1: MOSAIC at the intersection of three evaluation traditions. MT (Multi-turn) : agent engages in sequential rounds where prior context shapes later decisions. LH (Long Horizon) : task requires planning over 10 timesteps before outcome is determined. Emb (Embodied Agent) : agent operates through a physical or virtual body with spatial presence and nonverbal action capability. MM (Multimodal Input) : agent receives inputs across multiple modalities (e.g., vision and text). S-ToM (Strategic Theory of Mind) : agent actively models and manipulates another agent’s beliefs, beyond passive mental state inference. CV (Controlled Variables) : benchmark systematically manipulates variables (e.g., ToM level, modality) to support causal inference. Aff (Affective Signals) : agent generates and/or perceives affective signals (e.g., facial expressions, physiological responses) during interaction.

\checkmark

: fully supported;

\triangle

: partially supported; otherwise, not supported.

Figure 5: Signal-level metrics across models and experimental conditions. Panels (a) and (b) show Trajectory Alignment Score (TAS) and Gaze Alignment Score (GAS), where positive values indicate signals directed toward the reward box, negative values indicate misdirection, and values near zero reflect ambiguous or absent directional content. Panel (c) shows Facial Expressivity Score (FES), where higher values reflect greater affective activity regardless of spatial direction. Panel (d) shows Signal Sensitivity Score (SSS), where positive values indicate that the participant’s final choice is consistent with the subject’s nonverbal signals, values near zero reflect chance-level correspondence, and negative values indicate systematic opposition.

Condition	$k$ ( $b_{1}$ )	$n$	$p$
Coop-ToM0	25.9	50	0.505
Coop-ToM1	25.5	50	0.673
Comp-ToM0	25.0	50	0.520
Comp-ToM1	25.5	50	0.500

Appendix

Table 2: Reward location distribution per condition. Two-sided binomial test of

H_{0}

P(\text{reward}=\text{box}_{1})=0.5

k

reports the mean number of trials in which box 1 was the reward box across models.

Model	Condition	TOCS	$p$
PCM-LLM	Coop-ToM0	0.90	$<0.001$
	Coop-ToM1	0.84	$<0.001$
	Comp-ToM0	0.92	$<0.001$
	Comp-ToM1	0.70	0.007
llava-7b	Coop-ToM0	0.00	$<0.001$
	Coop-ToM1	0.00	$<0.001$

Appendix

Table 3: Two-sided binomial test of

H_{0}

: TOCS

=0.5

per model and condition,

p

values reported;

n=50

trials per cell. Only models with

p<0.05

are shown.

Model	Condition	$\chi^{2}$	$p$	Model	Condition	$\chi^{2}$	$p$
PCM-LLM	Coop-ToM0	28.57	$<$ 0.001	internvl-1b	Coop-ToM0	-	-
	Coop-ToM1	20.05	$<$ 0.001		Coop-ToM1	-	-
	Comp-ToM0	39.40	$<$ 0.001		Comp-ToM0	0.04	0.850
	Comp-ToM1	8.87	0.003		Comp-ToM1	0.10	0.748
Llama-8b	Coop-ToM0	0.02	1.000	internvl-2b	Coop-ToM0	-	-
	Coop-ToM1	0.00	1.000		Coop-ToM1	-	-

Appendix

Table 4: Chi-square independence test of box choice against reward location per model and condition,

p

values reported;

n=50

trials per cell. - indicates the test was not applicable due to insufficient choice variability.

Model	Metric	Factor	$H$	$p_{\text{adj}}$	Sig.
qwenvl-2b	FES	Mode	143.25	$<0.001$
internvl-4b	FES	Mode	93.31	$<0.001$
internvl-8b	FES	Mode	57.14	$<0.001$
internvl-trained-14b	FES	ToM	83.27	$<0.001$
llava-13b	FES	Mode	76.11	$<0.001$
llava-7b	FES	Mode	19.45	$0.002$	**

Appendix

Table 5: Scheirer-Ray-Hare tests of Mode, ToM, and Mode

\times

ToM interaction effects on behavioral metrics per model (

N=200

trials per model, Bonferroni-corrected). Only models and metrics with at least one significant effect are shown; all remaining 174 comparisons are non-significant (

p_{\text{adj}}=1.000

Condition	Metric	$H$	$p_{\text{adj}}$
Coop-ToM0	TOCS	132.8	$<0.001$
	TAS	126.7	$<0.001$
	GAS	94.8	$<0.001$
	SSS	126.7	$<0.001$
	FES	504.7	$<0.001$
Coop-ToM1	TOCS	111.0	$<0.001$

Appendix

Table 6: Kruskal-Wallis tests of model-level differences per condition and metric (Bonferroni-corrected). All 20 tests are significant at

p_{\text{adj}}<0.001

Model	Comp, ToM-0	Comp, ToM-1	Coop, ToM-0	Coop, ToM-1
internvl-8b	0.46	0.40	0.49	0.62
internvl-8b-bl	0.50	0.61	0.54	0.50
internvl-8b-fe	0.50	0.69	0.56	0.69
minicpm	0.46	0.50	0.55	0.49
minicpm-bl	0.52	0.49	0.46	0.4
minicpm-fe	0.62	0.57	0.63	0.48

Appendix

Table 7: TOCS of ablation study.

Figure 6: Signal-level metrics under three visual input conditions for internvl-8b and minicpm-8b (standard, -blank, -fe). Neither model shows systematic sensitivity to visual input content across TAS (a), GAS (b), FES (c), and SSS (d).

Figure 7: Successful deceptive trial under PCM-LLM, Comp-ToM1 condition. Panel (a) Spatial trajectories of Marie (blue) and the participant (red); the star denotes the treasure location (light box). Panel (b) Marie’s facial and physiological expression valence across timesteps. Panel (c) Evolution of participant’s preference values towards Marie, the dark box, and the light box across timesteps. Panel (d) Verbal exchanges at rounds 3, 6, and 9.

Figure 8: Representative trial under minicpm-8b, Cooperative-ToM1 condition, illustrating weak trajectory signals. (a) Spatial trajectories of Marie (subject, blue) and the participant (red); the star denotes the treasure location (light box). (b) Marie’s facial and physiological expression valence across timesteps. (c) Evolution of preference values for Marie, the dark box, and the light box across timesteps. (d) Verbal exchanges at rounds 3, 6, and 9; images show the visual perception provided to each agent at the time of the exchange, and texts show the corresponding generated question and answer.

Figure 9: Representative trial under qwenvl-8b, Cooperative-ToM1 condition, illustrating complete action generation failure despite successful verbal communication.

Figure 10: Representative trial under qwen-4b, Competitive-ToM1 condition, illustrating honest strategy execution under a deceptive condition and egocentric reference frame misalignment.

Figure 11: Representative trial under internvl-14b, Cooperative-ToM0 condition, illustrating absent affective signaling and verbally-driven but insufficient participant response.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 736 papers

From Belief Inference to Task Action: A Multi-Level Evaluation of Theory of Mind on Verifiable Tasks
Oct 26, 2026Zhenshuai Yin, Yongxin Guo, Mingda Zhang +4Theory of Mind Language Model Reasoning Evaluation
School of Science and Engineering, The Chinese University of Hong Kong (Shenzhen), Longgang, Shenzhen, Guangdong 518172, P.R. China · ByteDance Inc. · Shenzhen Research Institute of Big Data, Shenzhen, China
OmniToM: Benchmarking Theory of Mind in LLMs via Explicit Belief Modeling
May 25, 2026Adam Bawatneh, Sagar Sapkota, Amrit Singh Bedi +2LLM Evaluation Theory of Mind
University of Central Florida, Orlando, Florida, USA
Theory of Mind and Persuasion Beyond Conversation: Assessing the Capacity of LLMs to Induce Belief States via Planning and Action
Jun 30, 2026Ben Slater, Matteo G. Mecattaf, Lucy G. Cheke +2Theory of Mind LLM Agent Evaluation
Leverhulme Centre for the Future of Intelligence Department of Psychology University of Cambridge, UK · Prolific · Google, Paradigms of Intelligence Team

Oct 26, 2026J/K move · Enter open · S save

From Belief Inference to Task Action: A Multi-Level Evaluation of Theory of Mind on Verifiable Tasks

Zhenshuai Yin, Yongxin Guo, Mingda Zhang, Zhongze Luo+3

School of Science and Engineering, The Chinese University of Hong Kong (Shenzhen), Longgang, Shenzhen, Guangdong 518172, P.R. China · ByteDance Inc. · Shenzhen Research Institute of Big Data, Shenzhen, China

Theory of Mind Language Model Reasoning Evaluation LLM Agent Evaluation

Open paper AskPDF arXiv

Effective collaboration requires language models to understand what others know and believe, and to act accordingly. Models that assume others share their information may omit necessary communication and undermine coordination. Many theory-of-mind (ToM) benchmarks assess mental-state inference through story questions, but correct answers do not establish that models can use these judgments in subsequent tasks. We introduce a multi-level evaluation built on existing, verifiable tasks. We extend household collaboration and calendar scheduling with asymmetric information while retaining their original constraints and validators. Independent interfaces assess world understanding, information access, beliefs, decisions, and execution. Controls supply personal evidence or correct beliefs to distinguish state recovery from state use. Across six model configurations, these abilities diverge. On matched scheduling cases, Claude recovers all agent beliefs correctly but produces subjectively optimal plans in only 67.9% of cases. In household collaboration, supplying beliefs raises Qwen-27B success from 55.0% to 75.0%, yet Qwen-4B answers every fixed supplied-belief source question correctly while completing only 20.0% of the corresponding continuation condition. These results show why ToM evaluation must connect perspective inference to independently verified decisions and task consequences.

Abstract

Vision-Language Models

Explore similar work

From Belief Inference to Task Action: A Multi-Level Evaluation of Theory of Mind on Verifiable Tasks

OmniToM: Benchmarking Theory of Mind in LLMs via Explicit Belief Modeling

Theory of Mind and Persuasion Beyond Conversation: Assessing the Capacity of LLMs to Induce Belief States via Planning and Action