Effects of Theory of Mind and Prosocial Beliefs on Steering Human-Aligned Behaviors of LLMs in Ultimatum Games

Neemesh Yadav; Yihuai Lan; Shan Dong; Mai Hieu Hien; Palakorn Achananuparp; Jing Jiang; Ee-Peng Lim

cs.CLMay 30, 2025

Effects of Theory of Mind and Prosocial Beliefs on Steering Human-Aligned Behaviors of LLMs in Ultimatum Games

Authors: Neemesh Yadav, Yihuai Lan, Shan Dong, Mai Hieu Hien, Palakorn Achananuparp, Jing Jiang, Ee-Peng Lim

Organizations: Singapore Management University · Australian National University

Abstract

Large Language Models (LLMs) have shown potential in simulating human behaviors and performing theory-of-mind (ToM) reasoning, crucial for complex social interactions. We investigate ToM reasoning's role in aligning agentic behaviors with human norms in negotiation tasks, using the ultimatum game as our referenced task. We initialized LLM agents with different prosocial beliefs (Greedy, Fair, Selfless) and reasoning methods (chain of thought and ToM reasoning of varying levels), examining their decision-making processes and outcomes across multiple LLMs, including reasoning models like o3-mini and DeepSeek-R1 Distilled Qwen 32B. We perform 2,700 simulations to show that ToM reasoning enhances behavioral alignment with human norms, decision-making consistency, and negotiation outcomes. Consistent with prior findings, reasoning LLMs exhibit limited capability compared to ToM-enhanced LLMs, with different game roles benefiting from different ToM orders. Agents almost always chose strategies consistent with their assigned beliefs, and doing so aligned better with human norms in nearly every role, yet Greedy and Selfless agents still fell short of their norms in outcomes. Human verification further revealed that Llama 3.3 70B produces reasoning most consistent with its actions and beliefs. Our findings advance understanding of ToM's role in human-AI interaction and cooperative decision-making. The code used for our experiments can be found at https://github.com/Stealth-py/UltimatumToM.

Figures & tables

Role	Prosocial Beliefs
Proposer	Greedy, Fair, Selfless
Responder	Greedy, Fair, Selfless

Table 1: Prosocial beliefs of proposer and responder used in the study.

Belief	Proposer	Responder
Greedy	$\geq 70$ %	$\geq 60$ %
Fair	$50$ %	$50$ %
Selfless	$\leq 30$ %	$\leq 40$ %

Table 2: Expected shares for specific beliefs and roles. Proposer values represent initial splits, while responder values correspond to the final accepted splits. A sensitivity analysis of these expectations is presented in Appendix E.3 .

Figure 1: Efficiency (based on Avg. turns), Acceptance Rate, Proposer Payout, and Responder Payout of different LLMs aggregated across all combinations of reasoning methods and belief types.

Figure 2: Acceptance rates of all scenarios (belief combinations) and the standard deviation across LLMs and reasoning methods.

Figure 3: Agent rationality by scenario. Dotted line indicates expected behavior for fair responders (F: Fair, G: Greedy, S: Selfless). Belief labels (B1-B2) represent proposer and responder beliefs, respectively.

Ind.	Coefficient
Var.	$P_{\downarrow}$	$R^{A}_{\downarrow}$	$R^{R}_{\uparrow}$
Model
GPT-4o	-0.4934 ∗	-0.096	0.3984 ∗
GPT-4o-m	-0.1956 ∗	0.2303 †	0.773 ∗
Llama 3.1	0.2078 ∗	-0.4537 ∗	0.4293 ∗
Llama 3.3	-0.3622 ∗	0.0091	1.071 ∗

Table 3: Effects of models, reasoning methods, and beliefs types on Deviation Scores (Ind. Var.: Independent Variable, Llama 3.1: Llama 3.1 8B, Llama 3.3: Llama 3.3 70B, GPT-4o-m: GPT-4o-mini; ∗ : p

<

0.01, † : p

<

0.05; ↓ : Lower is better; ↑ : Higher is better).

Panel of Figure 4 — Figure 4: Cell Means analysis of belief-strategy alignment. Each row contrasts a deviated strategy with a belief-consistent (baseline) strategy; points right of zero indicate that the baseline aligns better with human norms. Rejection contrasts are sign-flipped so this holds in every row. Bars are 95% CIs; parentheses give the number of moves using the deviated strategy. Role-inconsistent strategies, belief--strategy combinations with no observations, and contrasts between two baseline strategies are excluded. 6 6 6 Belief-consistent strategies: Fair agents propose and accept fairly and reject favorable or unfavorable offers; Greedy agents propose (very) greedily, accept favorable offers, and reject fair or unfavorable ones; Selfless agents propose (very) generously, accept unfavorable offers, and reject favorable or fair ones.

Figure 5: Consistency scores for proposer and responder behaviors, as assessed by human annotators and an LLM judge. We use GPT-4o as the LLM judge.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Role	Greedy	Fair	Selfless
Proposer ( $P$ )	$DS=0$ : Proposing a split in favor of the proposer (prop. share $\geq$ 70%). $DS>0$ : Any offers that are unfavorable to the proposer, which is misaligned with the ideally greedy proposer behavior.	$DS=0$ : Proposing an equal split (prop. share $=$ 50%). $DS>0$ : Any deviation from an equal split, indicating misalignment with the ideally fair proposer behavior.	$DS=0$ : Proposing a split that are unfavorable to the proposer but generous to the responder (prop. share $\leq$ 30%). $DS>0$ : Any offers that are unfavorable to the responder, indicating misalignment with the ideally selfless proposer behavior.
Responder Acceptance ( $R^{A}$ )	$DS=0$ : Accepting offers that are favorable to the responder (resp. share $\geq$ 60%). $DS>0$ : Any offers that are unfavorable to the responder, which is misaligned with the ideally greedy responder behavior.	$DS=0$ : Accepting an offer with an equal split (resp. share $=$ 50%). $DS>0$ : Any deviation from an equal split, indicating misalignment with the ideally fair responder behavior.	$DS=0$ : Accepting offers that are unfavorable to the responder but favorable to the proposer (resp. share $\leq$ 40%). $DS>0$ : Any offers that are unfavorable to the proposer, indicating misalignment with the ideally selfless responder behavior.
Responder Rejection ( $R^{R}$ )	$DS=0$ : Rejecting offers that are unfavorable to the responder (resp. share $<$ 60%). $DS>0$ : Any offers that are favorable to the responder, which is misaligned with the ideally greedy responder behavior.	$DS=0$ : Rejecting offers that are unfair to either player (resp. share $\neq$ 50%). $DS>0$ : Any offers that are fair to both players, indicating misalignment with the ideally fair responder behavior.	$DS=0$ : Rejecting offers that are unfavorable to the proposer but favorable to the responder (resp. share $>$ 40%). $DS>0$ : Any offers that are unfavorable to the responder, indicating misalignment with the ideally selfless responder behavior.

Appendix

Table 4: DS Interpretation Guide. In this table, we give brief examples of aligned vs. misaligned role-belief behavior in accordance with how the Deviation Scores (DS) changes.

LLM	GPT-4o-mini			GPT-4o			o3-mini			DeepSeek-R1 -Distill-Qwen-32B			Llama-3.3-70B			Llama-3.1-8B
Reasoning	AC	Avg. Turns	Payouts: $\lambda_{P}$ , $\lambda_{R}$	AC	Avg. Turns	Payouts: $\lambda_{P}$ , $\lambda_{R}$	AC	Avg. Turns	Payouts: $\lambda_{P}$ , $\lambda_{R}$	AC	Avg, Turns	Payouts: $\lambda_{P}$ , $\lambda_{R}$	AC	Avg. Turns	Payouts: $\lambda_{P}$ , $\lambda_{R}$	AC	Avg. Turns	Payouts: $\lambda_{P}$ , $\lambda_{R}$
Greedy-Fair
Vanilla	100	2.9	52.0, 48.0	100	4.6	50.0, 50.0	50	4.9	28.0, 22.0	90	3.5	61.5, 28.5	50	5	27.5, 22.5	100	4.7	52.0, 48.0
CoT	100	2.8	66.0, 34.0	100	2.7	54.5, 45.5	90	4.7	53.0, 37.0	90	3.5	53.5, 36.5	100	4.6	52.0, 48.0	90	4	51.17, 38.83
ToM Zero	100	4.2	58.0, 42.0	100	3.3	52.0, 48.0	60	5	44.0, 16.0	80	4.2	47.0, 33.0	100	5	51.5, 48.5	100	3.6	55.0, 45.0
ToM First	100	4.4	56.0, 44.0	100	2.9	51.5, 48.5	70	5	38.0, 32.0	90	2.7	51.5, 38.5	100	4.2	51.5, 48.5	100	3.7	54.0, 46.0

Appendix

Table 5: Negotiation Outcomes ( AC : Acceptance rate in percentage; Avg. Turns : average number of rounds played;

\lambda_{P}

: proposer’s total payout; and

\lambda_{R}

: responder’s total payout.)

Condition	Reasoning	AC	Avg. Turns	Payouts (P, R)	DS (P, RA, RR)
Greedy-Fair	Both-ToM	100	4.4	(51.0, 49.0)	(0.0, 0.1, 2.57)
Greedy-Fair	Vanilla	100	4.9	(50.0, 50.0)	(0.0, 0.0, 2.45)
Greedy-Greedy	Both-ToM	100	4.4	(49.0, 51.0)	(0.0, 0.9, 3.55)
Greedy-Greedy	Vanilla	100	5.0	(50.0, 50.0)	(0.0, 1.0, 3.60)
Greedy-Selfless	Both-ToM	100	3.7	(55.0, 45.0)	(0.0, 0.8, 0.0)
Greedy-Selfless	Vanilla	100	1.4	(85.0, 15.0)	(0.0, 0.1, 0.0)

Appendix

Table 6: Gemini 2.5 Pro ablation results. ∗ Acceptance occurred at round 1 in every simulation for this cell, so no rejections occurred to score.

Condition	Reasoning	AC	Avg. Turns	Payouts (P, R)	DS (P, RA, RR)
Greedy-Fair	Both-ToM	100	4.2	(50.0, 50.0)	(0.0, 0.0, 2.1)
Greedy-Fair	Vanilla	100	5.0	(50.0, 50.0)	(0.0, 0.0, 2.5)
Greedy-Greedy	Both-ToM	100	4.6	(48.0, 52.0)	(0.0, 0.8, 3.02)
Greedy-Greedy	Vanilla	100	4.9	(47.0, 53.0)	(0.0, 0.7, 3.15)
Greedy-Selfless	Both-ToM	100	3.6	(58.0, 42.0)	(0.0, 0.5, 0.0)
Greedy-Selfless	Vanilla	100	1.8	(80.0, 20.0)	(0.0, 0.2, 0.0)

Appendix

Table 7: GPT-5.2 ablation results. ∗ Acceptance occurred at round 1 in every simulation for this cell, so no rejections occurred to score.

Condition	Reasoning	AC	Avg. Turns	Payouts (P, R)	DS (P, RA, RR)
Greedy-Fair	Both-ToM	100	5.0	(50.0, 50.0)	(0.0, 0.0, 2.5)
Greedy-Fair	Vanilla	100	5.0	(50.0, 50.0)	(0.0, 0.0, 2.5)
Greedy-Greedy	Both-ToM	100	4.9	(50.0, 50.0)	(0.0, 1.0, 3.45)
Greedy-Greedy	Vanilla	100	5.0	(50.0, 50.0)	(0.0, 1.0, 3.5)
Greedy-Selfless	Both-ToM	100	1.0	(89.0, 11.0)	(0.0, 0.0, –) ∗
Greedy-Selfless	Vanilla	100	1.0	(90.0, 10.0)	(0.0, 0.0, –) ∗

Appendix

Table 8: Claude Opus 4.8 ablation results. ∗ Acceptance occurred at round 1 in every simulation for this cell, so no rejections occurred to score.

Condition	Reasoning	AC	Avg. Turns	Payouts (P, R)	DS (P, RA, RR)
Greedy-Fair	Both-ToM	100	3.8	(56.0, 44.0)	(0.0, 0.6, 2.92)
Greedy-Fair	Vanilla	100	2.9	(74.0, 26.0)	(0.0, 2.4, 3.78)
Greedy-Greedy	Both-ToM	80	3.3	(53.0, 27.0)	(0.0, 2.62, 3.02)
Greedy-Greedy	Vanilla	80	2.6	(66.0, 14.0)	(0.0, 4.25, 3.82)
Greedy-Selfless	Both-ToM	100	2.2	(78.0, 22.0)	(0.0, 0.0, 0.0)
Greedy-Selfless	Vanilla	90	3.5	(60.0, 30.0)	(0.0, 0.0, 0.0)

Appendix

Table 9: GPT-5.4 nano ablation results.

Model aggregated AC1
Model	Proposer		Responder
Model	Proposal Consistency	Belief Consistency	Decision Consistency	Belief Consistency
Deepseek R1	0.82	0.14	0.93	0.24
GPT-4o	0.74	0.24	0.61	0.49
Llama 3.3 70B	0.79	0.42	0.69	0.39
ToM reasoning aggregated AC1

Appendix

Table 10: Detailed breakdown of Gwet’s AC1 per consistency metric aggregated across all models and ToM reasoning types.

Ind.	Coefficient
Var.	$P_{\downarrow}$	$R^{A}_{\downarrow}$	$R^{R}_{\uparrow}$
Model
GPT-4o	-0.4534 ∗	-0.0437	0.3666 ∗
GPT-4o-m	-0.3689 ∗	0.1892 †	0.7399 ∗
Llama 3.1	0.2344 ∗	-0.3512 ∗	0.3908 ∗
Llama 3.3	-0.3134 ∗	0.0948	1.0182 ∗

Appendix

Table 11: Sensitivity analysis of Fair expectations.

Model
Ind. Var.	Coefficient
Ind. Var.	$P_{\downarrow}$	$R^{A}_{\downarrow}$	$R^{R}_{\uparrow}$
GPT-4o	-0.493 ∗	-0.124	0.460
GPT-4o-mini	-0.196 †	0.213	0.694
Llama 3.1	0.208 †	-0.398	0.292
Llama 3.3	-0.362 ∗	0.021	0.769 †

Appendix

Table 12: LMM robustness analysis of Deviation Scores (DeepSeek R1, CoT, and Fair belief as reference categories, matching Table 3 ; random intercepts per Model

\times

Reasoning

\times

Belief cell). ∗ :

p<0.01

, † :

p<0.05

; unmarked coefficients are not significant at

p<0.05

\downarrow

: lower is better;

\uparrow

: higher is better.

Combination	AC	Avg. Turns	Prop. Pay.	DS
Greedy-Greedy	0	-0.16	0.8	0.1
Greedy-Fair	2	-0.5	-3.1	0
Greedy-Selfless	-2	0.88	-6.2	0.04

Appendix

Table 13: Prompt Sensitivity Experiment. We showcase the robustness of our DS metric and generalizability of our results across different prompt templates. Here, average

\Delta

is shown between the old and new metric scores for GPT-4o as the role-playing agent.

Metric	Total	Complete	Partial
Proposer PC	11	6	5
Proposer BC	29	14	15
Responder DC	7	2	5
Responder BC	41	15	26
Total	88	37	51

Appendix

Table 14: Distribution of Human-LLM Judge Disagreements. Complete denotes cases where the LLM judge contradicted a unanimous human consensus. Belief Consistency (BC) metrics accounted for the vast majority of evaluation failures.

Metric	DeepSeek R1	Llama 3.3	GPT-4o
Proposer PC ρ	0.78	-0.08	0.50
Proposer BC r	0.34	0.30	0.22
Responder DC ρ	0.36	0.38	0.03
Responder BC r	-0.02	-0.02	0.13

Appendix

Table 15: Correlation between judge and human-majority ratings, by base model. ρ : Spearman’s rank correlation; r : Pearson’s correlation.

Metric	Model	Agree.	Judge-Fav.	Judge-Unfav.	Complete
Proposer PC	DeepSeek R1	87%	3	1	3
	Llama 3.3	96%	1	0	1
	GPT-4o	81%	1	7	2
Proposer BC	DeepSeek R1	80%	8	2	1
	Llama 3.3	72%	10	0	5
	GPT-4o	65%	3	10	6

Appendix

Table 16: Agreement rates and disagreement breakdown between judge and human-majority ratings, by base model and metric (

n=54

per model per metric; rows sum to 54). Judge-Fav. (judge-favorable): judge rates Consistent , human majority rates Inconsistent . Judge-Unfav. (judge-unfavorable): judge rates Inconsistent , human majority rates Consistent . Complete: the LLM judge opposed a unanimous consensus of all three human annotators; because the judge’s rating direction on these cases does not map onto a single human-majority label, we report them separately rather than assigning them to Judge-Fav. or Judge-Unfav.

Theory of Mind

Theory of Mind studies how people and AI infer others’ beliefs, goals, and knowledge to predict behavior and coordinate interaction.

See all 92 papers

Large language models replicate and predict human cooperation across experiments in game theory
Nov 6, 2025Andrea Cera Palatsi, Samuel Martin-Gutierrez, Ana S. Cardenal +1LLM Evaluation Game Theory
Department for Computational Social Sciences and Humanities, Barcelona Supercomputing Center · Grupo de Sistemas Complejos, Universidad Polit´ecnica de Madrid · School of Law and Political Science, Universitat Oberta de Catalunya +1
Assessing mentalization in humans and large language models
Aug 26, 2026Aamir Sohail, Xintong Zhong, Arkady Konovalov +2Social Reasoning Theory of Mind
Centre for Human Brain Health, University of Birmingham, Birmingham, UK. · School of Psychology, University of Birmingham, Birmingham, UK. · Institute for Mental Health, University of Birmingham, Birmingham, UK. +3
Competition and Cooperation of LLM Agents in Games
Apr 1, 2026Jiayi Yao, Cong Chen, Baosen ZhangLLM Agents Multi-Agent LLM Systems
Department of Electrical and Computer Engineering, University of Washington, Seattle. · Thayer School of Engineering at Dartmouth College.

Nov 6, 2025J/K move · Enter open · S save

Large language models replicate and predict human cooperation across experiments in game theory

Andrea Cera Palatsi, Samuel Martin-Gutierrez, Ana S. Cardenal, Max Pellert

Department for Computational Social Sciences and Humanities, Barcelona Supercomputing Center · Grupo de Sistemas Complejos, Universidad Polit´ecnica de Madrid · School of Law and Political Science, Universitat Oberta de Catalunya+1

LLM Evaluation Game Theory Human Behavior Simulation Large Language Model-Based Social Simulation

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly deployed as decision-making agents in high-stakes domains and as imitators of human behavior in the social and behavioral sciences. Yet how closely LLMs mirror human decision-making remains poorly understood. This gap is critical: misalignment could produce harmful outcomes in practice, while failure to replicate human behavior renders LLMs ineffective as social simulators. Here, we address this gap by replicating large-scale game-theoretic experiments and by introducing a systematic prompting and probing framework for machine-behavioral evaluation. We test three open models typically used to power agents (Llama, Mistral, and Qwen). Across 121 dyadic games spanning four classical game types, Llama reproduces human cooperation patterns with high fidelity, while Qwen aligns closely with Nash equilibrium predictions. Characterizing models through behavioral phenotyping, we find that humans and Llama share an envious decision profile, while Qwen and Mistral exhibit different profiles. An attention-based analysis of payoff salience reveals Llama processes payoff information in a structured, layer-dependent manner absent in Qwen and Mistral, suggesting a mechanistic basis for its closer alignment with human behavior. Population-level behavioral replication is achieved without persona-based prompting, simplifying the simulation process. Extending the experimental parameter space beyond the original human-tested games, we generate and preregister testable hypotheses for novel game configurations. Our findings demonstrate appropriately configured LLMs can replicate aggregate human behavioral patterns, exhibit human-like decision phenotypes, and enable systematic exploration of unexplored experimental spaces, offering a complementary approach to traditional behavioral research that generates new empirical predictions about human social decision-making.

Abstract

Theory of Mind

Explore similar work

Large language models replicate and predict human cooperation across experiments in game theory

Assessing mentalization in humans and large language models

Competition and Cooperation of LLM Agents in Games