PerSo: Investigating Facets of LLMs' Social Intelligence in Real-World Persuasion Scenarios

Ali Zamani; Soroush Esfahanian; Erfan Shahabi; Mohammad Taher Pilehvar; Azadeh Shakery; Yadollah Yaghoobzadeh

cs.CLOct 14, 2026

PerSo: Investigating Facets of LLMs' Social Intelligence in Real-World Persuasion Scenarios

Authors: Ali Zamani, Soroush Esfahanian, Erfan Shahabi, Mohammad Taher Pilehvar, Azadeh Shakery, Yadollah Yaghoobzadeh

Organizations: School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran · Cardiff University, United Kingdom · Institute for Research in Fundamental Sciences (IPM), Tehran, Iran · Tehran Institute for Advanced Studies, Khatam University, Tehran, Iran

Abstract

Understanding the social intelligence of large language models (LLMs) is increasingly important for applications involving human-AI interaction and multi-agent environments. Social intelligence comprises two aspects: cognitive intelligence, the ability to infer others' mental states, and behavioral intelligence, the ability to select appropriate social actions based on these inferences to achieve social goals. Persuasion provides a natural testbed in which both aspects are simultaneously at play, yet prior benchmarks typically evaluate them in isolation using synthetic scenarios. We introduce PerSo, a benchmark that evaluates both aspects on real-world persuasion dialogues from Reddit's ChangeMyView community, using an LLM-in-the-loop pipeline to construct the evaluation tasks, validated through human annotation. Our evaluation of diverse LLMs shows that performance generally scales with model size but varies across dimensions. The strongest LLM slightly surpasses the best observed human performance on the behavioral task, while the best human participant remains ahead on the cognitive task.

Figures & tables

Benchmark	Real Human Data	Cognitive	Cog. Aspects	Cog. Type	Behavioral
FaNToM ( Kim et al., 2023 )	✗	✓	B	R	✗
DYNToM ( Xiao et al., 2025 )	✗	✓	B, E, I, A	R	✗
PersuasiveToM ( Yu et al., 2025 )	✗	✓	B, D, I	R	✗
SocialEval ( Zhou et al., 2025 )	✗	✗	–	–	✓
SOTOPIA ( Zhou et al., 2023 )	✗	✗	–	–	✓
AgentSense ( Mou et al., 2025 )	✗	✗	–	–	✓

Table 1: Comparison of social intelligence benchmarks. B: Belief, D: Desire, E: Emotion, I: Intention/Intent, A: Action. R: mental-state recognition, where the target state is inferred from observed text; F: mental-state forecasting, where the target state is predicted before observing the subsequent utterance that manifests it.

Figure 1: Overview of the PerSo pipeline: data extraction and processing from CMV, multiple-choice question generation for cognitive intelligence, and the behavioral and cognitive intelligence evaluation tasks.

Dataset overview
Threads	200
Date range	Jan 2024–Oct 2025
Label balance (A / B)	100 / 100
Text length (words)	Mean	Median
OP initial post	449	400
Persuader response ( $+$ )	242	182

Table 2: Summary statistics of the PerSo dataset.

Participant	PerSo accuracy (%)				TSIS scores (%)
	Behavioral	Cognitive			Social Information Processing	Social Skills	Social Awareness
	Behavioral	Belief	Emotion	Intent	Social Information Processing	Social Skills	Social Awareness
A	65.0	85.0	65.0	85.0	79	61	71
B	40.0	50.0	52.5	60.0	54	54	50
C	75.0	75.0	62.5	72.5	50	75	64

Table 3: Participant-level PerSo accuracy and TSIS scores. PerSo results were obtained before gold-label annotation on 20 behavioral instances and 40 cognitive instances per dimension, following the protocol in Section 4.1.2 . TSIS is a self-report measure ( Silvera et al., 2001 ) . Bold indicates the highest value in each score column.

Dimension	A–B	A–C	B–C	Avg. $\kappa$
Belief	100.0% 1.000	90.0% 0.866	90.0% 0.866	0.911
Emotion	70.0% 0.601	70.0% 0.602	75.0% 0.667	0.623
Intent	82.5% 0.764	72.5% 0.630	80.0% 0.733	0.709

Table 4: Pairwise agreement (%, first line) and Cohen’s

\kappa

(second line) for cognitive answer selection with the final OP reply visible.

Model	Behavioral Intelligence	Cognitive Intelligence
Model	Behavioral	Belief	Emotion	Intent
Gemma-4B	60.5	50.0	20.5	54.5
Gemma-12B	62.5	58.25	23.25	63.25
Gemma-27B	71.0	60.0	25.75	64.5
Qwen-4B (W/O Thinking)	64.0	66.0	25.25	60.75
Qwen-35B (W/O Thinking)	75.0	71.75	35.75	73.75

Table 5: Behavioral and cognitive task accuracy (%). Bold marks the best model score in each column; underlining marks the best score within each multi-row model block, including ties, with Qwen thinking modes treated separately. Human (avg/best) reports the mean/columnwise maximum across three participants.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Dimension	Fleiss’ $\kappa$	Krippendorff’s $\alpha$
Belief	0.911	0.911
Emotion	0.622	0.625
Intent	0.708	0.711

Appendix

Table 6: Multi-annotator agreement across all three annotators (A, B, and C) jointly, using two complementary reliability metrics.

Dimension	Gold Correctness	Distractor Plausibility
Belief	4.775 $\pm$ 0.474	4.575 $\pm$ 0.628
Emotion	4.517 $\pm$ 0.866	4.258 $\pm$ 0.780
Intent	4.750 $\pm$ 0.520	4.558 $\pm$ 0.739
Overall	4.681 $\pm$ 0.650	4.464 $\pm$ 0.716

Appendix

Table 7: Human evaluation of gold answer correctness and distractor plausibility using a 5-point Likert scale, with the third utterance visible to annotators. Values are reported as mean

\pm

standard deviation.

Figure 2: Distribution of choice A versus choice B on the behavioral intelligence task, for each model and human annotator. The dagger symbol ( $\dagger$ ) denotes models evaluated with thinking enabled.

Figure 3: Absolute pairwise difference in %A between all evaluated models and human annotators on the behavioral intelligence task. The dagger symbol ( $\dagger$ ) denotes models evaluated with thinking enabled. Model proportions are computed over all instances, whereas human proportions are computed on the instances shown to the annotators; differences involving human annotators should therefore be interpreted with caution.

Figure 4: Distribution of choices A, B, C, and D for the belief dimension of the cognitive intelligence task, for each model and human annotator. The dagger symbol ( $\dagger$ ) denotes models evaluated with thinking enabled.

Figure 5: Pairwise Pearson correlation of answer-choice distributions between all evaluated models and human annotators, for the belief dimension of the cognitive intelligence task. Correlations involving human annotators are computed on shared instances. Blank cells indicate undefined correlations, which occur when a distribution computed on the shared instances is exactly uniform. The dagger symbol ( $\dagger$ ) denotes models evaluated with thinking enabled.

Figure 6: Distribution of choices A, B, C, and D for the emotion dimension of the cognitive intelligence task, for each model and human annotator. The dagger symbol ( $\dagger$ ) denotes models evaluated with thinking enabled.

Figure 7: Pairwise Pearson correlation of answer-choice distributions between all evaluated models and human annotators, for the emotion dimension of the cognitive intelligence task. Correlations involving human annotators are computed on shared instances. The dagger symbol ( $\dagger$ ) denotes models evaluated with thinking enabled.

Figure 8: Distribution of choices A, B, C, and D for the intent dimension of the cognitive intelligence task, for each model and human annotator. The dagger symbol ( $\dagger$ ) denotes models evaluated with thinking enabled.

Figure 9: Pairwise Pearson correlation of answer-choice distributions between all evaluated models and human annotators, for the intent dimension of the cognitive intelligence task. Correlations involving human annotators are computed on shared instances. The dagger symbol ( $\dagger$ ) denotes models evaluated with thinking enabled.

Topic	Count	%
Ethics & Morality	60	30.0
Politics & Government	45	22.5
Society & Culture	34	17.0
Philosophy & Religion	30	15.0
Economics & Business	9	4.5
Science & Technology	7	3.5

Appendix

Table 8: Topic distribution of the 200 CMV threads in PerSo .

Method	Accuracy (%)	$\pm$ Std
Behavioral task ( $n=200$ )
Random baseline	50.0	–
Length heuristic	65.5	–
Logistic Regression	68.0	8.9
Random Forest	65.5	4.3
Gradient Boosting	63.5	5.4

Appendix

Table 9: Non-LLM baseline results. Std is computed across the five cross-validation folds; deterministic methods have none.

Figure 12: Random Forest feature importance on the behavioral task. Each feature is the difference between its values for the two candidate replies.

Model	Full (200)	Filtered (167)
Gemma-4B	60.50	56.29
Gemma-12B	62.50	59.88
Gemma-27B	71.00	67.66
Qwen-4B	64.00	57.49
Qwen-35B	75.00	70.06
Qwen-397B	76.00	71.26

Appendix

Table 10: Behavioral accuracy (%) on the full dataset and on the filtered subset obtained by removing instances with an extreme length difference between the two replies (length ratio above Q3).

\dagger

denotes models evaluated with thinking enabled.

Figure 17: A processed example illustrating how a paired persuasion branch is converted into the cognitive and behavioral tasks.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,402 papers

$Ψ$ -Bench: Evaluating Persona-Sensitive Influencing in Persuasive Dialogues
Jun 1, 2026Peixuan Han, Hongyi Du, Jiayu Liu +3LLM Personalization LLM Agent Evaluation
University of Illinois Urbana-Champaign
LLM Persuasion Is in the Eye of the Evaluation
Oct 7, 2026Kamile Dementaviciute, Julija Vaitonyte, Tijl De BiePersuasion in Language Models LLM Evaluation
Ghent University · Tilburg University, Department of Computational Cognitive Science · ISM University of Management and Economics
AREG: Adversarial Resource Extraction Game for Evaluating Persuasion and Resistance in Large Language Models
Feb 18, 2026Adib Sakhawat, Fardeen SadabLLM Evaluation Persuasion in Language Models
Department of Computer Science and Engineering Islamic University of Technology, Dhaka, Bangladesh

Jun 1, 2026J/K move · Enter open · S save

$Ψ$ -Bench: Evaluating Persona-Sensitive Influencing in Persuasive Dialogues

Peixuan Han, Hongyi Du, Jiayu Liu, Yihang Sun+2

University of Illinois Urbana-Champaign

LLM Personalization LLM Agent Evaluation AI Agent Benchmarks Persuasion in Language Models

Open paper AskPDF arXiv

Personalization is a crucial capability of modern language agents. However, current research primarily positions personalized agents as passive responders to user preferences, limiting their ability to interact with users and provide suggestions or guidance proactively. To systematically evaluate such proactive personalization in realistic interactions, we propose

Ψ

-Bench, a benchmark for assessing LLMs' ability to influence realistic users through conversation. We design three real-world interaction scenarios that involve persuasion in

Ψ

-Bench, and endow simulated clients with personal characteristics through explicit user profiles derived from dialogue histories. We evaluate 10 frontier LLMs on

Ψ

-Bench and find that while most models can produce coherent and reasonable arguments, even state-of-the-art models still leave considerable room for improvement in persuasion. We also find that providing access to client profiles yields an average performance gain of 18.24%, highlighting the importance of user-specific information for effective persuasion. Overall, our work highlights persona-sensitive influencing as a challenging yet practical direction for evaluating and developing more proactive personalized LLM agents. Codes are available at: https://github.com/Hanpx20/Psi-Bench.

Abstract

LLM Evaluation

Explore similar work

ΨΨΨ-Bench: Evaluating Persona-Sensitive Influencing in Persuasive Dialogues

LLM Persuasion Is in the Eye of the Evaluation

AREG: Adversarial Resource Extraction Game for Evaluating Persuasion and Resistance in Large Language Models

$Ψ$ -Bench: Evaluating Persona-Sensitive Influencing in Persuasive Dialogues