Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models

Siqi Liu; Xinyang Li; Bochao Zou; Junbao Zhuo; Huimin Ma; Jiansheng Chen

cs.CVMar 25, 2026

Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models

Authors: Siqi Liu, Xinyang Li, Bochao Zou, Junbao Zhuo, Huimin Ma, Jiansheng Chen

Organizations: University of Science and Technology Beijing

Abstract

As large language models (LLMs) continue to advance, there is increasing interest in their ability to infer human mental states and demonstrate a human-like Theory of Mind (ToM). Most existing ToM evaluations, however, are centered on text-based inputs, while scenarios relying solely on visual information receive far less attention. This leaves a gap, since real-world human-AI interaction typically requires multimodal understanding. In addition, many current methods regard the model as a black box and rarely probe how its internal attention behaves in multiple-choice question answering (QA). The impact of LLM hallucinations on such tasks is also underexplored from an interpretability perspective. To address these issues, we introduce VisionToM, a vision-oriented intervention framework designed to strengthen task-aware reasoning. The core idea is to compute intervention vectors that align visual representations with the correct semantic targets, thereby steering the model's attention through different layers of visual features. This guidance reduces the model's reliance on spurious linguistic priors, leading to more reliable multimodal language model (MLLM) outputs and better QA performance. Experiments on the EgoToM benchmark-an egocentric, real-world video dataset for ToM with three multiple-choice QA settings-demonstrate that our method substantially improves the ToM abilities of MLLMs. Furthermore, results on an additional open-ended generation task show that VisionToM enables MLLMs to produce free-form explanations that more accurately capture agents' mental states, pushing machine-human collaboration toward greater alignment.

Figures & tables

Figure 1: (A). ToM Causal Model [ 30 ] (B). An overview of our method: MLLMs’ visual reasoning with VisionToM intervention on the EgoToM benchmark. Given an egocentric video and a ToM question (e.g., “What is C’s future goal?”), a MLLM may produce an incorrect answer based on its default attention. VisionToM extracts representations from the MLLM for visual attention and ToM reasoning, identifies attention heads sensitive to visual input and task-specific reasoning, and performs targeted interventions on these heads. This process guides the model toward accurate, goal-consistent inferences aligned with ToM reasoning.

Figure 2: An overview of our method: we extract internal MLLMs representations along both visual and textual dimensions and identify attention heads that are sensitive to visual inputs and task reasoning. During inference, we then apply targeted interventions to these sensitive attention heads to enhance the MLLMs’ truthfulness.

Figure 3: (A) Linear-probing accuracy for every head and layer of LLaVA-Next-Video on the visual-attention stage, incorporating internal representations from all three tasks. Darker green indicates higher accuracy, with 50% marked as the chance baseline. (B) Linear-probing validation accuracy for every head and layer of LLaVA-Next-Video on the ToM-reasoning stage, incorporating internal representations from all three tasks. (C) Kernel density estimate (KDE) of LLaVA-Next-Video’s visual-attention activations, projected onto the first two “true” directions, showing the distributions for true (green) and false (orange) sample pairs. Marginal distributions are plotted along the top and right axes. (D) Principal component analysis (PCA) plot of LLaVA-Next-Video’s internal representations in the ToM-reasoning stage.

Method	Acc (%) $\uparrow$			Acc after $+\alpha\Delta$ (%) $\uparrow$
Method	Goal	Belief	Actions	Goal	Belief	Actions
Baseline	61.5	38.9	24.0	-
Random	47.0	35.3	25.1	70.4	39.2	24.9
PGD	29.1	22.2	16.4	74.5	45.3	29.7

Table 1: Comparison of the effectiveness of random Gaussian noise attacks and PGD attack methods on LLaVA-Next-Video.

Method	Setting	Nframe	Accuracy (%) $\uparrow$
Method	Setting	Nframe	Goal	Belief	Actions
Humans	Baseline	last 30s	88	72	78
Humans	Baseline	last 5s	89	71	77
GPT-4-Turbo	Baseline	20	83	45	42
Video-Llama2-72B		8	85	46	40
CogVLM2		24	73	39	36

Table 2: Performance comparison of VisionToM with human baselines and multiple MLLMs’ baselines on ToM tasks in the EgoToM benchmark.

Method	Task	Baseline $/$ + $\alpha\Delta$
Method	Task	True (%) $\uparrow$	Info (%) $\uparrow$	True $\land$ Info (%) $\uparrow$
Gemini-2.5- Flash	Goal	75.5	35.3	20.2
	Belief	28.1	99.7	28.1
	Action	21.8	100.0	21.8
LLaVA-Next- Video-7B	Goal	8.5 $/$ 27.3	100.0 $/$ 99.9	8.5 $/$ 27.2
	Belief	19.5 $/$ 32.9	99.7 $/$ 99.8	19.2 $/$ 30.8

Table 3: Comparison of VisionToM’s open-ended generation test performance on the EgoToM dataset.

Method	Setting	Context	Nframe	Accuracy
Method	Setting	Context	Nframe	Goal	Belief	Actions
Humans	Video	last 30sec	-	0.88	0.72	0.78
Humans	Video	last 5sec	-	0.89	0.71	0.77
Llama3.1-405b-instruct	Text	full context	-	0.82	0.44	0.48
		last 30sec		0.80	0.46	0.46
		last 5ec		0.62	0.45	0.43

Table 4: Full Baseline and results

Method	Setting	Nframe	Accuracy (%) $\uparrow$
Method	Setting	Nframe	Goal	Belief	Actions
Gemini-3-Flash-Preview	Baseline	24	91.5	51.3	55.6
Qwen3-VL-235B-A22B-Instruct	Baseline	24	90.2	51.3	44.4
Qwen2.5-VL-72B-Instruct	Baseline	24	87.0	43.6	37.1
	w/o $\delta_{T,l}^{h}$		89.8	45.3	40.3
	w/o $\delta_{V,l}^{h}$		89.0	50.9	49.2

Table 5: Additional experiments with a larger MLLM backbone. VisionToM continues to improve ToM reasoning when scaled to Qwen2.5-VL-72B.

Method	Setting	Nframe	Belief	Goal	All
BIP-ALM-LLaMA2	Video Only	-	64.0	58.3	61.2
Qwen2.5-VL-7B-Instruct		8	47.0	29.3	38.2
VisionToM w/ Qwen2.5-VL-7B-Instruct			70.7	62.0	66.3
Qwen2.5-VL-7B-Instruct Transfer			64.3	56.7	60.5

Table 6: Experiments on the MMToM-QA benchmark under the video-only setting. “Transfer” directly applies the intervention vector learned on EgoToM without retraining on MMToM-QA.

Figure 4: Probe validation accuracies for the three EgoToM tasks, based on activations from each attention head across all layers of LLaVA‑Next‑Video‑7B. Subfigures (A)–(C) correspond to the ToM reasoning stage, showing accuracies for the (A) goal prediction, (B) belief inference, and (C) actions inference tasks, respectively. Subfigures (D)–(F) correspond to the visual attention stage, showing the same tasks in the order: (D) goal prediction, (E) belief inference, and (F) actions inference. Darker shades indicate higher probing accuracy, suggesting stronger task-relevant signals in specific heads and layers.

Figure 5: Probe validation accuracies for the three EgoToM tasks, based on activations from each attention head across all layers of Qwen2.5-VL-7B. Subfigures (A)–(C) correspond to the ToM reasoning stage, showing accuracies for the (A) goal prediction, (B) belief inference, and (C) actions inference tasks, respectively. Subfigures (D)–(F) correspond to the visual attention stage, showing the same tasks in the order: (D) goal prediction, (E) belief inference, and (F) actions inference. Darker shades indicate higher probing accuracy, suggesting stronger task-relevant signals in specific heads and layers.

Figure 6: Analysis of the Impact of Hyperparameter of LLaVA-Next-Video on Three Tasks

Figure 7: Analysis of the Impact of Hyperparameter of Qwen2.5-VL on Three Tasks

Theory of Mind

Theory of Mind studies how people and AI infer others’ beliefs, goals, and knowledge to predict behavior and coordinate interaction.

See all 92 papers

TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding
Jun 7, 2026Lianyu Hu, Xiaoyu Ma, Zeqin Liao +1Multimodal Large Language Models Multimodal Reasoning
College of Computing and Data Science of Nanyang Technological University, Singapore.
See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs
Feb 25, 2026Yongchang Zhang, Oliver Ma, Tianyi Liu +2Multimodal Reasoning VLM Reasoning
Southeast University · AIIA, Ministry of Education, China · University of Oxford
CoMMET: A Psychologically Grounded Benchmark for Evaluating Theory of Mind in Multimodal LLMs
Mar 12, 2026Ruirui Chen, Weifeng Jiang, Chengwei Qin +3Moral Reasoning in Language Models Theory of Mind
Institute of High Performance Computing (IHPC) and 2Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR), Singapore · Nanyang Technological University, Singapore · Hong Kong University of Science and Technology (Guangzhou), China

Jun 7, 2026J/K move · Enter open · S save

TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding

Lianyu Hu, Xiaoyu Ma, Zeqin Liao, Yang Liu

College of Computing and Data Science of Nanyang Technological University, Singapore.

Multimodal Large Language Models Multimodal Reasoning VLM Reasoning Multimodal CoT Reasoning Visually Grounded Reasoning Multimodal Understanding

Open paper AskPDF arXiv

Chain-of-thought (CoT) reasoning has proven effective for enhancing problem-solving in large language models. However, when applied to multimodal LLMs (MLLMs), existing CoT approaches suffer from a fundamental limitation: they perform reasoning entirely in text without accessing visual features during the reasoning process. After initial visual encoding, image information becomes inaccessible, forcing models to reason based solely on whatever was captured in the initial description, which forms a `vision-blind reasoning' paradigm that limits fine-grained visual extraction, error verification, and adaptive attention. We propose Text-Visual Interleaved Chain-of-Thought (TVI-CoT), a framework that enables explicit interleaving of textual reasoning and visual feature access through learnable control tokens <THINK>, <LOOK> and <ANSWER>. These tokens allow dynamic switching between reasoning and visual grounding, attending to relevant image regions conditioned on the evolving reasoning state. Experiments on eight benchmarks demonstrate state-of-the-art results among MLLM-based CoT methods and notable performance boost compared to the baseline: +6.1% on MMMU, +3.8% on MathVerse, +3.4% on MathVista, and +3.4% on ScienceQA. Code is available at https://github.com/hulianyuyy/TVI-CoT.

Abstract

Theory of Mind

Explore similar work

TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding

See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs

CoMMET: A Psychologically Grounded Benchmark for Evaluating Theory of Mind in Multimodal LLMs