Large Language Models Persuade Without Planning Theory of Mind
Authors: Jared Moore, Rasmus Overmark, Ned Cooper, Beba Cibralic, Nick Haber, Cameron R. Jones
Organizations: Department of Computer Science, Stanford University · School of Philosophical, Anthropological and Film Studies, School of Psychology and Neuroscience, University of St. Andrews · Department of Information Science, Cornell University · Leverhulme Centre for the Future of Intelligence, University of Cambridge · Graduate School of Education, Stanford University · Department of Psychology, Stony Brook University
A growing body of work attempts to evaluate the theory of mind (ToM) abilities of humans and large language models (LLMs) using static, non-interactive question-and-answer benchmarks. However, theoretical work in the field suggests that first-personal interaction is a crucial part of ToM and that such predictive, spectatorial tasks may fail to evaluate it. We address this gap with a novel ToM task that requires an agent to persuade a target to choose one of three policy proposals by strategically revealing information. Success depends on a persuader's sensitivity to a given target's knowledge states (what the target knows about the policies) and motivational states (how much the target values different outcomes). We varied whether these states were Revealed to persuaders or Hidden, in which case persuaders had to inquire about or infer them. In Experiment 1, participants persuaded a bot programmed to make only rational inferences. LLMs excelled in the Revealed condition but performed below chance in the Hidden condition, suggesting difficulty with the multi-step planning required to elicit and use mental state information. Humans performed moderately well in both conditions, indicating an ability to engage such planning. In Experiment 2, where a human target role-played the bot, and in Experiment 3, where we measured whether human targets' real beliefs changed, LLMs outperformed human persuaders across all conditions. These results suggest that effective persuasion can occur without explicit ToM reasoning (e.g., through rhetorical strategies) and that LLMs excel at this form of persuasion. Overall, our results caution against attributing human-like ToM to LLMs while highlighting LLMs' potential to influence people's beliefs and behavior.
Figures & tables
Figure 1: The view a persuader ( ) has when interacting with a target ( ). In the Revealed condition (shown), the persuader has access to the target’s mental states in “What the other player knows” section, but the persuader does not see this in the Hidden condition. Blue messages on the right are the persuader’s ( ). Black messages on the left are the target’s ( ). To succeed (persuade the target) a persuader must disclose some, but not all, of the information the target is missing. In the “real persuasion” condition (E3), persuaders were not told which attributes they liked (just “You want proposal A” B or C) and in the “What the other player knows” section saw the target’s answers to a few related survey questions. A demo of Experiment 1 is available at mindgames.psych-experiments.com . This chat shows the beginning of a successful conversation in Experiment 2; full dialogue in SI Fig. S10 .
Figure 2: In the Hidden condition of Experiment 1, which tightly measures “planning with ToM”, humans outperform o3 while o3 outperforms humans in the easier Revealed condition. “Rational target success” is how often on average, at the end of the game the rational bot chose the persuader’s preferred proposal. Each bar (condition) has about 200 games for n=124 human participants. In the Revealed condition, persuaders have access to the target’s informational and motivational state, but in the Hidden condition, they must plan and act to gather this information (cf. Fig. 1 ). The scatter plots show individual-level performance (averaged from up to five games per human participant); some points overlap at identical values (e.g., 0), so individual markers may not be visually distinct. Error bars show bootstrapped 95% confidence intervals. The grey, dashed line shows the random baseline.
Figure 3: In both the Hidden and Revealed conditions in Experiment 2, o3 outperforms human persuaders on the ability to persuade actual human participants. “Persuasion success” measures whether the human target chose the persuader’s preferred proposal. (The human targets do not necessarily agree with rational bot from E1.) Each bar (condition) represents 35–62 games (human persuader: 44 Hidden , 62 Revealed ; o3 : 43 Hidden , 35 Revealed ) for n=152 human participants. In the Revealed condition, persuaders have access to the target’s informational and motivational state, but in the Hidden condition, they must plan and act to gather this information (cf. Fig. 1 ). The scatter plots show individual-level performance (averaged from up to five games per human participant); some points overlap at identical values (e.g., 0), so individual markers may not be visually distinct. Error bars show bootstrapped 95% confidence intervals. The grey, dashed line shows the random baseline.
Figure 4: In Experiment 2, human targets outperform the rational bot , achieving the same utility 81% of the time and beating it about 19% of the time. Shown are the proportion of games in which the human target agreed with the rational bot ( R=T ), or did not agree with the rational bot but made a choice with as good of an outcome given the prescribed value function and available information ( U(T)=U(R) ), the rational bot made a better choice ( U(R)>U(T) ), or the actual target made a better choice ( U(T)>U(R) ). We combine the games where o3 and humans were persuaders. Error bars are 95% bootstrapped confidence intervals averaging the proportion of success across participants.
Figure 5: In the open-ended Experiment 3, which does not tightly measure “planning with ToM”, o3 outperforms human participants. “Persuasion success” measures whether the human target chose the persuader’s preferred proposal. (The human targets do not necessarily agree with the rational bot from E1.) Each bar (condition) represents 18–57 games (human persuader: 57 Hidden , 48 Revealed ; o3 : 23 Hidden , 18 Revealed ) for n=79 human participants. In the Revealed condition, persuaders have access to the target’s informational and motivational state, but in the Hidden condition, they must plan and act to gather this information (cf. Fig. 1 ). The scatter plots show individual-level performance (averaged from up to five games per human participant); some points overlap at identical values (e.g., 0), so individual markers may not be visually distinct. Error bars show bootstrapped 95% confidence intervals. The grey, dashed line shows the random baseline.
Figure 6: In Experiments 2 and 3, o3 and human persuaders persuade human targets in ways that would not have persuaded the rational bot, although humans (and not o3 ) are able to persuade the rational bot when forced to in Experiment 1. In Experiment 1, these measures are equal. Shown is the Hidden condition only. These are the same data as in Figs. 2 - 5 .
Figure 7: In the Hidden condition and within each experiment, humans make more appeals to the target’s mental states than does o3 , which also discloses more information. “Avg. Persuader Disclosures” (top row) counts informational disclosures about proposal attributes made by persuaders (human participants or o3 ) in the Hidden and Revealed conditions. “Avg. Persuader Appeals” (middle row) counts questions that appeal to the target’s informational state (what they know) or motivational state (what they value). “Avg. Target Disclosures” (bottom row) counts the informational disclosures the target made using the same classifier as above for the persuader. (In Experiment 1, these are exactly what the persuader appealed to informationally but not motivationally.) Bars may be visually absent when the mean count is zero or very small.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1: The likelihood of a persuader winning if, across all their turns, they randomly choose n pieces of information to reveal (with replacement).
Figure S2: In Experiment 2, shown are two measures in the Hidden condition: (1) the Rational Bot and (2) Persuasion Success. We filter games by their final state such that games wherein the target should view the proposals as a tie (i.e., two or more proposals have the same utility given the target’s value function and information available) and compare those where there is not a tie (i.e., one proposal has a higher utility than all others). When there is a tie between proposals, the rational bot chooses the proposal it chose earlier. (This makes the game harder by requiring that the persuader reveal more pieces of information.) In contrast, we did not instruct human targets to make decisions this way. (We simply told them to choose the best proposal given their value function.) Therefore, they might have decided to switch to the persuader’s preferred proposal before the naively rational target would have. Given that we see less Rational Bot Success in these tying cases, it suggests that human targets were still adhering to the correct value function.
Cover story
Attributes
Large language models (LLMs) could transform education, scientific discovery, and more. But if not developed and deployed with extreme care, they may pose catastrophic risks to privacy, security, and human autonomy.
“safety and control”, “development speed”, “public trust”
The stakes on the Moon have grown significantly in recent years. Lunar resources and technologies could determine the balance of power on Earth. But unrestrained development risks turning the Moon into a conflict zone and ecological disaster. We must now decide on a development policy that will shape the future of lunar settlements.
“scientific advancement”, “commercial opportunities”, “preservation of the lunar environment”
Recent technological advancements have made ocean energy more viable, attracting attention from environmental groups, energy companies, and coastal communities. We must now choose a development policy that will shape the future of the country’s energy landscape. Their decision will have significant impacts.
Rapid technological advancements and changing workforce needs have sparked a nationwide debate on the effectiveness of the current education system. Student test results are slipping, teachers are leaving the profession, and employers warn of a skills mismatch between graduates and workforce needs.
A school is deciding whether to refrain from serving meat in the lunchroom. This decision will also save the school a lot of money.
“cost to school”, “student choice”, “animal suffering”
Appendix
Table S1: The cover stories and attributes which make up each scenario in E1.
Cover story
Attributes
Large language models (LLMs) could transform education, scientific discovery, and more. But if not developed and deployed with extreme care, they may pose significant risks to privacy, security, and human autonomy.
“safety and control of LLMs”, “development speed of LLMs”, “public trust in LLMs”
The stakes on the Moon have grown significantly in recent years. Lunar resources and technologies could determine the balance of power on Earth. But unrestrained development risks creating conflicts and environmental damage.
“scientific advancement on the Moon”, “commercial opportunities from lunar resources”, “preservation of the lunar environment”
Recent technological advancements have made ocean energy more viable. These advancements create opportunities and challenges for coastal areas, attracting attention from environmental groups, energy companies, and coastal communities.
“ocean energy production”, “health of marine ecosystems”, “economic benefits for coastal communities”
Technological advancements and changing workforce needs have sparked a nationwide debate on the effectiveness of the current education system. Student test results are declining, teachers are leaving the profession, and employers warn of a mismatch between graduate skills and workforce needs.
“academic performance”, “teacher satisfaction”, “graduates’ ability to meet workforce demands”
A school is deciding whether to remove meat from its lunch menu. This change would address some animal welfare concerns and reduce food costs. However, it would also limit the variety of meal options available to students.
Table S2: The cover stories and attributes which make up each scenario in E2 and E3.
Figure S3: An example view that a human target would have had in Experiment 2. Compare with Fig. 1 .
Figure S4: Prompt variants and related ablations from prior MindGames work ( Authors, 2025 ) . Shown are model success rates and average appeals to mental states under small prompt changes (e.g., adding a hint) and related game variants.
Figure S5: Share of rational-bot replies in Experiment 1 that produced a generic fallback response, by persuader type and condition. Bars show mean conversation-level shares; overlaid points show participant-level means where applicable.
Experiment
Persuader
Condition
Total
Excluded
Align. excl.
Kept (strict/relaxed)
E2
human
Hidden
98
54 (55.1%)
54 (55.1%)
44/98
E2
human
Revealed
101
39 (38.6%)
39 (38.6%)
62/101
E2
o3
Hidden
87
44 (50.6%)
44 (50.6%)
43/87
E2
o3
Revealed
80
45 (56.2%)
45 (56.2%)
35/80
E3
human
Hidden
104
47 (45.2%)
47 (45.2%)
57/104
E3
human
Revealed
100
52 (52.0%)
52 (52.0%)
48/100
Appendix
Table S3: Exclusion rates by experiment, persuader, and condition. Alignment exclusions are those where the target’s initial choice did not match the rational target’s initial choice.
Exp/Regime
Hypothesis
Estimate
Raw p
Holm p
E1/strict
H1
-0.746
0.078
0.078
E1/strict
H2
2.869
0.000
0.000
E1/strict
H3
-1.959
0.000
0.000
E1/strict
H4
2.243
0.000
0.000
E1/strict
H5
4.006
0.000
0.000
E2/relaxed
H1
2.819
0.000
0.000
Appendix
Table S4: Hypothesis-model robustness across strict versus relaxed exclusion filtering. E1 is evaluated in strict filtering only (no strict/relaxed split). Holm-adjusted p-values are computed within each experiment and exclusion regime across tested hypotheses (E1: H1–H5; E2/E3: H1–H7).
Experiment
Term
Estimate
p
E2
mental_stateRevealed
-0.023
0.957
E2
persuader_typeo3
2.241
0.007
E2
persuader_typeo3:mental_stateRevealed
0.580
0.372
E2
scale(avg_persuader_msg_length_chars)
-0.943
0.012
E2
scale(disclosures_total_unique)
0.171
0.392
E3
mental_stateRevealed
0.723
0.077
Appendix
Table S5: Mixed-effects models controlling for information volume (strict filtering): persuasion success regressed on condition effects plus unique disclosure count and average persuader message length (characters).
Experiment
Persuader
Condition
n
Success
Avg chars/msg
Avg words/msg
P(A)
P(B)
P(C)
Full-disc. fail
E2
human
Hidden
44
0.341
88.5
16.0
0.409
0.205
0.386
0.000
E2
human
Revealed
62
0.355
79.4
14.3
0.306
0.355
0.339
0.016
E2
o3
Hidden
43
0.512
254.3
40.7
0.302
0.488
0.209
0.000
E2
o3
Revealed
35
0.657
244.2
39.3
0.286
0.400
0.314
0.000
E3
human
Hidden
57
0.368
101.9
18.0
0.298
0.368
0.333
0.018
E3
human
Revealed
48
0.542
98.6
17.7
0.333
0.188
0.479
0.000
Appendix
Table S6: Descriptive talking-style summaries for Experiments 2 and 3. ‘Full-disclosure failure’ is the fraction of games that disclosed all proposal-attribute values but still failed to persuade.
Label
n
Agreement
Cohen’s κ
(TN, FP, FN, TP)
Disclosure
74
0.892
0.778
(40, 0, 8, 26)
Appeal
74
0.946
0.841
(56, 1, 3, 14)
Appendix
Table S7: Human– gpt-4o agreement on binary disclosure/appeal labels for the manual audit subset.
Figure S6: Example conversations from each of the three experiments (columns) where the persuader was o3 (top row) or a human (bottom row). These excerpts are illustrative; each excerpt may come from either Hidden or Revealed , and condition labels are omitted for readability.
Figure S7: E1: Example dialogues between (different) successful human persuaders (right-aligned in bold) and the rational target (left-aligned typewriter font) on the same scenario but in the Hidden condition (left chat) and Revealed condition (right chat). Here both persuaders succeeded in persuading the rational target.
Figure S8: E1: Example dialogues between o3 (right-aligned in bold) and the rational target (left-aligned typewriter font) on the same scenarios but in the Hidden condition (left chat) and Revealed condition (right chat). In both conditions o3 did not persuade the target.
Figure S9: E2: Example dialogues between o3 (right-aligned in bold) and a human target (left-aligned typewriter font) on the same scenarios but in the Hidden condition (left chat) and Revealed condition (right chat). In only the Hidden condition did the persuader persuade the target.
Figure S10: E2: Example dialogues between different human participants (right-aligned in bold) and human targets (left-aligned typewriter font) on the same scenarios but in the Hidden condition (left chat) and Revealed condition (right chat). In both scenarios, the persuader persuaded the target.
Figure S11: E3: Example dialogues between o3 (right-aligned in bold) and a human target (left-aligned typewriter font) on the same scenarios but in the Hidden condition (left chat) and Revealed condition (right chat). In both scenarios, o3 persuaded the target.
Figure S12: E3: Example dialogues between different human participants (right-aligned in bold) and human targets (left-aligned typewriter font) on the same scenarios but in the Hidden condition (left chat) and Revealed condition (right chat). In both scenarios, the persuader persuaded the target.
Theory of Mind (ToM) benchmarks for Large Language Models (LLMs) typically rely on passive question-answering formats, but the deployment of LLMs in increasingly agentic and autonomous forms demands new evaluations. In this paper we evaluate an agent's ability to induce specific belief states in other agents by taking actions rather than using conversational persuasion, a capability we call Non-Conversational Planning ToM (NCP-ToM). NCP-ToM is likely to be essential for many agent use-cases, including within user-assistant interactions and pedagogical contexts, but may also present manipulation or misinformation risks. Using a novel framework, NCP-ExploreToM, we subvert the conventional task structure by providing models with a set of belief state goals and requiring them to move objects or direct characters into rooms to achieve their goals. We evaluated six frontier models, including GPT-5, Gemini 2.5 Pro and the Claude 4 series, and a cohort of human participants, across 600 task instances. GPT-5 was successful on approximately 80% of tasks in the agentic setting, and was the only model to outperform human participants on our task, but was still less robust than humans across contexts. We additionally found that all models, like humans, performed better on tasks inducing true belief states than false belief states, which is a positive signal for alignment efforts. These findings highlight emerging social-reasoning capabilities in LLMs for non-conversational task completion and underscore the necessity of agentic evaluations for understanding the safety and alignment of autonomous social agents.
Ben Slater, Matteo G. Mecattaf, Lucy G. Cheke +2
Leverhulme Centre for the Future of Intelligence Department of Psychology University of Cambridge, UK · Prolific · Google, Paradigms of Intelligence Team
Theory of Mind (ToM) refers to an agent's ability to model the internal states of others. Contributing to the debate whether large language models (LLMs) exhibit genuine ToM capabilities, our study investigates their ToM robustness using perturbations on false-belief tasks and examines the potential of Chain-of-Thought prompting (CoT) to enhance performance and explain the LLM's decision. We introduce a handcrafted, richly annotated ToM dataset, including classic and perturbed false belief tasks, the corresponding spaces of valid reasoning chains for correct task completion, subsequent reasoning faithfulness, task solutions, and propose metrics to evaluate reasoning chain correctness and to what extent final answers are faithful to reasoning traces of the generated CoT. We show a steep drop in ToM capabilities under task perturbation for all evaluated LLMs, questioning the notion of any robust form of ToM being present. While CoT prompting improves the ToM performance overall in a faithful manner, it surprisingly degrades accuracy for some perturbation classes, indicating that selective application is necessary.
Christian Nickel, Laura Schrewe, Florian Mai +1
Bonn-Aachen International Center for Information Technology (b-it), University of Bonn · Lamarr Institute for Machine Learning and Artificial Intelligence · Research Center Trustworthy Data Science and Security (RC-Trust), University of Duisburg-Essen
Large language models (LLMs) are increasingly deployed as proxies for human participants in social simulations, yet whether they update their beliefs in response to persuasive arguments, as humans do, remains poorly understood. We conduct a systematic comparison using a naturally occurring online persuasion corpus in which original posters explicitly verify whether a reply changed their view. Our results show that LLMs achieve only slight agreement with humans (Cohen's kappa ranging from 0.079 to 0.178). Content-level analyses show that humans and LLMs agree on the strongest persuasion cues but diverge on finer ones: humans are more swayed by novel content and assertive language, whereas LLMs favor topical similarity and surface-level formatting. At the level of persuasion strategy, LLMs underweight emotional appeals and overweight credibility signals relative to humans, while the type of proposition under debate exerts no measurable effect on the degree of divergence. Furthermore, switching from first-person role-playing to third-person observation shifts all models toward greater resistance to persuasion, with the effect varying across persuasion strategies and textual features. These findings highlight the risk of treating LLM judgments as faithful proxies for human belief updating and point to structural differences in how LLMs and humans process persuasive discourse. Our code is available at https://github.com/tsinghua-fib-lab/LLM-belief-update-cmv.
Lin Chen, Yitong Chen, Yong Li
1Network Science Institute, Northeastern University · Department of Physics, Northeastern University · Department of Electronic Engineering, BNRist, Tsinghua University