Understanding Artificial Theory of Mind: Perturbed Tasks and Reasoning in Large Language Models
Authors: Christian Nickel, Laura Schrewe, Florian Mai, Lucie Flek
Organizations: Bonn-Aachen International Center for Information Technology (b-it), University of Bonn · Lamarr Institute for Machine Learning and Artificial Intelligence · Research Center Trustworthy Data Science and Security (RC-Trust), University of Duisburg-Essen
Theory of Mind (ToM) refers to an agent's ability to model the internal states of others. Contributing to the debate whether large language models (LLMs) exhibit genuine ToM capabilities, our study investigates their ToM robustness using perturbations on false-belief tasks and examines the potential of Chain-of-Thought prompting (CoT) to enhance performance and explain the LLM's decision. We introduce a handcrafted, richly annotated ToM dataset, including classic and perturbed false belief tasks, the corresponding spaces of valid reasoning chains for correct task completion, subsequent reasoning faithfulness, task solutions, and propose metrics to evaluate reasoning chain correctness and to what extent final answers are faithful to reasoning traces of the generated CoT. We show a steep drop in ToM capabilities under task perturbation for all evaluated LLMs, questioning the notion of any robust form of ToM being present. While CoT prompting improves the ToM performance overall in a faithful manner, it surprisingly degrades accuracy for some perturbation classes, indicating that selective application is necessary.
Figures & tables
Figure 2: Illustrating “Conclusion from Sentiment”.
Figure 3: Dataset illustration with per-sentence gold belief states shown inline. The belief column encodes what Esther believes about the container contents after each sentence.
Figure 4: Examples of reasoning chains compared to the gold CoT. In (a), the model outputs a valid proper subsequence with consistent reasoning; in (b), intermediate states are inconsistent or skipped (step 3 ”D” in the gold chain), leading to an invalid chain.
Perturbation Class
Llama 2 70B Chat
Vicuna 33B v1.3
Mixtral 8 × 7B Inst.
Yi 34B Chat
Llama 3 70B Inst.
DBRX Instruct
Average (all models)
V-P
CoT
ATE
V-P
CoT
ATE
V-P
CoT
ATE
V-P
CoT
ATE
V-P
CoT
ATE
V-P
CoT
ATE
V-P
CoT
ATE
No Perturbation
61.5
76.9
+15.4
27.3
81.8
+54.6
92.9
78.6
-14.3
42.9
100.0
+57.1
100.0
100.0
+0.0
78.6
100.0
+21.4
67.2
89.6
+22.4
Overall
46.6
46.6
+0.0
42.7
49.6
+6.8
47.4
50.4
+3.0
46.5
48.8
+2.3
62.5
69.9
+7.4
53.7
57.5
+3.7
49.9
53.8
+3.9
Transparent Container
28.6
28.6
+0.0
69.2
53.9
-15.4
7.1
35.7
+28.6
42.9
28.6
-14.3
42.9
57.1
+14.3
42.9
57.1
+14.3
38.9
43.5
+4.6
Preposition Replacement
20.0
0.0
-20.0
40.0
30.0
-10.0
20.0
20.0
+0.0
60.0
30.0
-30.0
20.0
40.0
+20.0
37.5
25.0
-12.5
32.9
24.2
-8.8
Uninformative Label
12.5
0.0
-12.5
37.5
25.0
-12.5
50.0
25.0
-25.0
25.0
12.5
-12.5
25.0
37.5
+12.5
62.5
62.5
+0.0
35.4
27.1
-8.3
Table 1: Accuracy (as percentages %) per perturbation class and model, comparing Vanilla prompting (V-P) vs Chain-of-Thought (CoT). Underline denotes performance above 50%, and bold denotes the best model performance on the data subset. ATE reports the signed absolute change (CoT − V-P), with improvements in green and deteriorations in red . We additionally report the Average over all models.
Statistic Type
Measure
Llama 2 70B Chat
Vicuna 33B v1.3
Mixtral 8x7B Inst.
Yi 34B Chat
Llama 3 70B Inst.
DBRX Instruct
Point- Biserial Correlation
ROUGE-LCSP
0.391
0.516
0.304
0.245
0.398
0.235
ROUGE-LCPSP
0.390
0.482
0.242
0.212
0.370
0.214
Transition Overlap
0.476
0.501
0.281
0.229
0.435
0.224
Phi Coefficient
CoT Correctness
0.549
0.342
0.429
0.306
0.584
0.489
Table 2: Faithfulness: Correlations between CoT-correctness and Final Answer Correctness for all models. Moderate to strong correlations suggest that models rely on their rationales, although other effects are present. P-values for these correlations are below the P = 0.05 threshold.
Figure 5: Comparison of effect strengths of CoT prompting given (a) incorrect (left-hand side) and (b) correct (right-hand side) reasoning rationales. Among all the evaluated models, we observe a placebo effect only in Mixtral, where incorrect reasoning (right) can even have a larger positive effect on final answer correctness than correct reasoning chains.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Table 5: Theory of Mind Accuracies: All Models and Perturbation Classes. Upper table created using ”All-Subtasks”. Lower table measurements on ”False Belief” Tasks. V-P: Vanilla - Prompting; CoT-P: Chain of Thought - Prompting.
Model
meta-llama/Llama-2-70b-chat-hf
lmsys/vicuna-33b-v1.3
mistralai/Mixtral-8x7B-Instruct-v0.1
01-ai/Yi-34B-Chat
meta-llama/Meta-Llama-3-70B-Instruct
databricks/dbrx-instruct
Effect
ATE
RR
ATE
RR
ATE
RR
ATE
RR
ATE
RR
ATE
RR
CoT
Perturbation Class
Statistic
Without CoT
Overall
meas.
-0.040
0.951
-0.037
0.956
-0.089
0.890
-0.037
0.955
-0.108
0.887
-0.099
0.887
CI95%
[-0.119, 0.038]
[0.864, 1.047]
[-0.118, 0.044]
[0.868, 1.053]
[-0.171, -0.006]
[0.802, 0.987]
[-0.116, 0.042]
[0.866, 1.052]
[-0.157, -0.060]
[0.843, 0.934]
[-0.169, -0.029]
[0.817, 0.962]
Transparent Container
meas.
-0.025
0.969
0.013
1.015
-0.091
0.887
0.003
1.003
-0.077
0.920
-0.011
0.988
CI95%
[-0.133, 0.082]
[0.849, 1.106]
[-0.094, 0.119]
[0.895, 1.151]
[-0.208, 0.025]
[0.759, 1.036]
[-0.104, 0.109]
[0.881, 1.142]
[-0.154, -0.000]
[0.845, 1.001]
[-0.105, 0.083]
[0.886, 1.100]
Appendix
Table 7: Perturbation Effect Strengths: All Models. All-Subtasks. ATE: Average Treatment Effect; RR: Relative Risk.
Model
meta-llama/Llama-2-70b-chat-hf
lmsys/vicuna-33b-v1.3
mistralai/Mixtral-8x7B-Instruct-v0.1
01-ai/Yi-34B-Chat
meta-llama/Meta-Llama-3-70B-Instruct
databricks/dbrx-instruct
Effect
ATE
RR
ATE
RR
ATE
RR
ATE
RR
ATE
RR
ATE
RR
CoT
Perturbation Class
Statistic
Without CoT
Overall
meas.
-0.149
0.758
0.155
1.567
-0.454
0.511
0.037
1.085
-0.378
0.622
-0.248
0.684
CI95%
[-0.398, 0.099]
[0.498, 1.153]
[-0.100, 0.409]
[0.638, 3.849]
[-0.643, -0.266]
[0.396, 0.657]
[-0.208, 0.281]
[0.616, 1.911]
[-0.529, -0.226]
[0.518, 0.747]
[-0.471, -0.025]
[0.504, 0.928]
Transparent Container
meas.
-0.330
0.464
0.420
2.538
-0.857
0.077
0.000
1.000
-0.571
0.429
-0.357
0.545
CI95%
[-0.649, -0.010]
[0.198, 1.088]
[0.090, 0.749]
[0.997, 6.461]
[-1.097, -0.618]
[0.007, 0.830]
[-0.325, 0.325]
[0.469, 2.133]
[-0.834, -0.309]
[0.247, 0.743]
[-0.666, -0.048]
[0.300, 0.991]
Appendix
Table 9: Perturbation Effect Strengths: All Models. False-Belief-Tasks. ATE: Average Treatment; Effect RR: Relative Risk.
Table 12: Chain of Thought Effect Strengths: All Models and Perturbation Classes. Upper table created using ”All-Subtasks”. Lower table measurements on ”False Belief” Tasks. ATE: Average Treatment Effect; RR: Relative Risk.
Statistic Type
Measure
Metric
Llama 2 70B Chat
Vicuna 33B v1.3
Mixtral 8x7B Inst.
Yi 34B Chat
Llama 3 70B Inst.
DBRX Instruct
Point- Biserial Correlation
ROUGE-L
Corr.
0.391
0.516
0.304
0.245
0.398
0.235
P-Value
2\times10−6
0
3.14\times10−4
4\times10−3
2\times10−6
5.989\times10−3
ROUGE-L (Pre-Proper)
Corr.
0.39
0.482
0.242
0.212
0.37
0.214
P-Value
3\times10−6
0
4.613\times10−3
1.3389\times10−2
1\times10−5
1.2283\times10−2
Transition Overlap
Corr.
0.476
0.501
0.281
0.229
0.435
0.224
P-Value
0
0
9.35\times10−4
7.429\times10−3
0
8.75\times10−3
Appendix
Table 13: Faithfulness: Correlations and P-Values between CoT-correctness and Final Answer Correctness for all models. Moderate to strong correlations suggest that models rely on their rationales, although other effects are present.
Figure 6: Structure of our dataset. Each of the 7 stages contains one unperturbed base task and up to 10 perturbed variants. Each task includes 16 subtasks probing different types of understanding.
Figure 7: ToM evaluation and analysis pipeline. Our dataset is used to benchmark six open-source LLMs under Vanilla and Chain-of-Thought prompting. Model answers are evaluated for final answer accuracy, CoT correctness, and faithfulness.
Large language models (LLMs) have recently shown strong performance on Theory of Mind (ToM) tests, prompting debate about the nature and validity of the underlying capabilities. At the same time, reasoning-oriented LLMs trained via reinforcement learning with verifiable rewards have demonstrated notable improvements across a range of benchmarks. In this work, we examine the behavior of such reasoning models in ToM tasks using novel adaptations of machine psychological experiments together with results from established benchmarks. We observe that reasoning models consistently exhibit increased robustness to prompt variations and task perturbations. Our analysis suggests these gains come at least partly from models being more robust at reaching the correct answer under prompt and task variation. We read this as evidence for a robustness-based account rather than for a new ToM-specific ability.
Ian B. de Haan, Peter van der Putten, Max van Duijn
The study explores whether current Large Language Models (LLMs) exhibit Theory of Mind (ToM) capabilities -- specifically, the ability to infer others' beliefs, intentions, and emotions from text. Given that LLMs are trained on language data without social embodiment or access to other manifestations of mental representations, their apparent social-cognitive reasoning raises key questions about the nature of their understanding. Are they capable of robust mental-state attribution indistinguishable from human ability in its output, or do their outputs merely reflect superficial pattern completion? To address this question, we tested five LLMs and compared their performance to that of human controls using an adapted version of a text-based tool widely used in human ToM research. The test involves answering questions about the beliefs, intentions, and emotions of story characters. The results revealed a performance gap between the models. Earlier and smaller models were strongly affected by the number of relevant inferential cues available and, to some extent, were also vulnerable to the presence of irrelevant or distracting information in the texts. In contrast, GPT-4o demonstrated high accuracy and strong robustness, performing comparably to humans even in the most challenging conditions. This work contributes to ongoing debates about the cognitive status of LLMs and the boundary between genuine understanding and statistical approximation.
Anna Babarczy, Andras Lukacs, Peter Vedres +1
Institute of General and Hungarian Linguistics, Research Centre for Linguistics · Department of Cognitive Science, Faculty of Natural Sciences, Budapest University of Technology and Economics · Institute of Mathematics, Eötvös Lóránd University
A growing body of work attempts to evaluate the theory of mind (ToM) abilities of humans and large language models (LLMs) using static, non-interactive question-and-answer benchmarks. However, theoretical work in the field suggests that first-personal interaction is a crucial part of ToM and that such predictive, spectatorial tasks may fail to evaluate it. We address this gap with a novel ToM task that requires an agent to persuade a target to choose one of three policy proposals by strategically revealing information. Success depends on a persuader's sensitivity to a given target's knowledge states (what the target knows about the policies) and motivational states (how much the target values different outcomes). We varied whether these states were Revealed to persuaders or Hidden, in which case persuaders had to inquire about or infer them. In Experiment 1, participants persuaded a bot programmed to make only rational inferences. LLMs excelled in the Revealed condition but performed below chance in the Hidden condition, suggesting difficulty with the multi-step planning required to elicit and use mental state information. Humans performed moderately well in both conditions, indicating an ability to engage such planning. In Experiment 2, where a human target role-played the bot, and in Experiment 3, where we measured whether human targets' real beliefs changed, LLMs outperformed human persuaders across all conditions. These results suggest that effective persuasion can occur without explicit ToM reasoning (e.g., through rhetorical strategies) and that LLMs excel at this form of persuasion. Overall, our results caution against attributing human-like ToM to LLMs while highlighting LLMs' potential to influence people's beliefs and behavior.
Jared Moore, Rasmus Overmark, Ned Cooper +3
Department of Computer Science, Stanford University · School of Philosophical, Anthropological and Film Studies, School of Psychology and Neuroscience, University of St. Andrews · Department of Information Science, Cornell University +3