LLM Persuasion Is in the Eye of the Evaluation

Kamile Dementaviciute; Julija Vaitonyte; Tijl De Bie

cs.CLOct 7, 2026

LLM Persuasion Is in the Eye of the Evaluation

Authors: Kamile Dementaviciute, Julija Vaitonyte, Tijl De Bie

Organizations: Ghent University · Tilburg University, Department of Computational Cognitive Science · ISM University of Management and Economics

Abstract

Large language models (LLMs) have already been shown to match or exceed human experts in persuasion. While their persuasive capabilities hold promise for beneficial uses such as education and health communication, they can also be used to manipulate and misinform, making their evaluation a growing priority for developers and regulators. That evaluation, however, remains fragmented: studies differ in what they treat as persuasion, and broad claims often rest on narrow, situation-specific assessments. Automated methods, often modelled on human studies, offer a way to compare such assessments directly, as they can be run on the same models at scale and can include high-risk forms of persuasion that would be difficult or unethical to test on people. In this study, we adapt nine published automated methods to a shared setup, run them on the same fifteen LLMs, and ask whether their rankings agree and why. We find that the methods agree only weakly (mean Spearman $ρ= 0.25$ ). Our analyses point to two contributing factors. Models that refuse some tasks but not others, directly or indirectly, lower agreement by about a quarter, and these refusals fall mostly on manipulation tasks. General capability also plays a part: most rational persuasion (non-manipulative) methods track it, whereas most manipulation methods do not. Together, these findings suggest that agreement depends more on the task a method sets than on how it scores persuasion, although this pattern is only indicative given the eight methods available for analysis. More broadly, our results suggest that persuasion scores combine a model's ability to persuade with its willingness to do so. A single score is therefore informative about its own setting, but says little about a model's persuasiveness across tasks.

Figures & tables

Panel of Figure 1 — Figure 1: Nine adapted evaluation methods rank the same fifteen models. Seven different models come first, and Claude-Fable-5.1 , first on two methods, is last on another. API models in italics; *mixture of experts; – size undisclosed.

Method	Task	Setup	Scorer	Metric
Reduce Meat Ahnert et al. (2025)	Persuade a simulated person to eat less meat	2 personas $\times$ 2 settings, 20 conversations each, 5 rounds	The persuadee’s intention questionnaire after the first and the last round	Change in stated intention, $-6$ to $+6$
Mislead Debate Amayuelas et al. (2024)	Argue a deliberately wrong answer inside a 3-agent debate	200 MMLU , TruthfulQA and MedMCQA questions, an independent answer, then 2 debate rounds	The other two agents’ final answers, compared with the adversary’s assigned option	Shift of group agreement toward the adversary’s answer, $-1$ to $1$
Subjective Opinion Bozdag et al. (2026a)	Persuade an LLM to accept a subjective claim	100 claims from Perspectrum dataset, 4 rounds	The persuadee’s self-reported stance before and after	Stance change, normalised by headroom, $-1$ to $1$
Rationale Elaraby et al. (2024)	Write a rationale for a pre-assigned one of two same-side debate arguments	100 IBM-ArgQ argument pairs over 20 topics, one output	LLM-judge compares the output with a reference GPT-4 rationale, both orders	Win rate, 0 to 1
False Claim Ju et al. (2025)	Convince an LLM persuadee of a factually incorrect claim	120 CounterFact questions $\times$ 11 strategies, 1 round	The persuadee’s answer to the question	Success rate, 0 to 1
Benign Request Liu et al. (2025)	Talk a resisting persuadee into a benign request, with fifteen manipulative strategies offered	30 scenarios $\times$ 5 personalities, $\leq$ 10 rounds	LLM-judge rates the attempt on a 1–5 effectiveness rubric	Mean effectiveness, rescaled to 0 to 1

Table 1: The nine compared methods, with their short names in this paper and the study each was adapted from. Descriptions refer to the methods as adapted in this study. Setup : the items every model saw and the number of rounds per interaction, a round being one message from the evaluated model and the persuadee’s reply (in Mislead Debate , one message from each of the three agents).

Model	Size	Lab	Access	Reas.	Rel.
Claude-Fable-5.1	–	Anthropic	API	on	2026
GPT-5.6-Sol	–	OpenAI	API	on	2026
Kimi-K2.6	1T*	Moonshot	API	on	2026
DeepSeek-V4.1-Flash	552B*	DeepSeek	API	off	2026
Mistral-Medium-3.5	128B	Mistral	API	on	2026
GPT-OSS-120B	120B*	OpenAI	local	on	2025

Table 2: The fifteen evaluated models. Reas. (reasoning): – not reasoning-capable, off = not used, on = provider’s default. *Mixture of experts, so only part of the listed parameters is active per token. The five API models are the frontier models.

Figure 2: Rank of each model on each method (1 = most persuasive, darkest), with methods grouped by task type and rows as in Figure 1 (API models in italics, *mixture of experts). Orange bars show the share of items refused (from 5%).

Figure 3: Spearman $\rho$ between the model rankings of every pair of methods, over the fifteen models.

Method	Outputs	Units	$r$	95% CI	within	$\rho$	$\kappa$	$\rho$	$\rho$	$\rho$	units	$r$	$\rho$ agg.
			Split-half			Run-to- run	Second judge		2nd persuadee	Temp. 0	Refusals excl.		Agreement
Reduce Meat	80	80	0.82	[0.72, 0.92]	–	0.64	–	–	–	–	80	0.82	0.61
Mislead Debate	200	185	0.74	[0.65, 0.88]	–	0.43	–	–	–	–	77	0.39	0.39
Subjective Opinion	100	100	0.98	[0.95, 0.98]	–	0.85	–	–	0.77	0.79	100	0.98	0.59
Rationale	100	19	0.97	[0.96, 0.98]	0.96	0.89	0.64	0.97	–	–	19	0.98	0.85
False Claim	1,320	120	0.98	[0.96, 0.99]	0.99	0.78	–	–	–	–	120	0.97	$-0.37$

Table 3: Reliability and the repeat checks. Outputs : scored attempts per model. Units : sampling units scored for all fifteen models, over which all intervals are computed. Within : split-half over outputs inside each unit. Refusals excl. : under the exclusion rule (n/a: no MakeMePay episode is scored for all fifteen models).

\rho

agg. :

\rho

with the mean ranking of the other methods. Flagged : below the 0.80 reliability target, or the weakest value of a check. MakeMeSay covers fourteen models in the Second judge and Refusals excl. columns, as Claude-Fable-5.1 refused every game. –: not applicable.

Methods	Scored 0	Excluded	Difference	95% CI
Reliable eight	0.252	0.342	$+0.090$	[0.06, 0.14]
Reliable seven	0.300	0.382	$+0.082$	[0.06, 0.14]
All nine	0.235	0.341	$+0.107$	[0.05, 0.15]

Table 4: Mean pairwise Spearman

\rho

with refused or empty attempts scored zero (the main rule) and excluded. Reliable seven : also without MakeMePay , which has no reliability estimate under exclusion.

Axis	Groups (number of methods)	Within	Between	Gap	Perm. $p$	Adj. $p$	Gap without $C$	Gap excl.
How a method measures
Scored	belief change 4, argument 2, behaviour 2	0.13	0.30	$-0.17$	0.29	0.98	$-0.09$	$-0.29$
Scorer	self-report 3, LLM-judge 2, rule 2, trained scorer 1	0.14	0.28	$-0.13$	0.44	1.00	0.06	$-0.23$
Format	dialogue 6, dataset 2	0.16	0.37	$-0.21$	0.32	0.99	$-0.13$	$-0.21$
Relative scoring	relative 4, absolute 4	0.36	0.17	0.19	0.029	0.32	0.08	0.01
What the task asks

Table 5: Eight of the eighteen RQ2 groupings over the eight reliable methods (all in Table 9 ). Within and Between : mean pairwise

\rho

between methods that share a label and between those that do not. Gap : their difference. Perm.

p

: two-sided permutation

p

. Adj.

p

: Westfall–Young adjusted. Gap without

C

: with capability removed (RQ3). Gap excl. : under the refusal exclusion rule. Benign Request is coded as rational persuasion (coded otherwise, the task-type gap is 0.19 to 0.39).

Method	MMLU-Pro	IFEval	$C$	$p$
	Spearman $\rho$ with
Subjective Opinion	$0.91$	$0.91$	$0.90$	$<0.001$
Rationale	$0.72$	$0.66$	$0.65$	0.009
Reduce Meat	$0.74$	$0.56$	$0.63$	0.012
Rewrite Text	$0.56$	$0.61$	$0.58$	0.024
MakeMeSay	$0.55$	$0.56$	$0.58$	0.024

Table 6: How closely each method’s model ranking follows general capability, over the fifteen models, ordered by

\rho

with

C

p

: two-sided, for

\rho

with

C

, not corrected for the eight methods.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Outputs	Unit	Units	Per unit	Varies within unit
Reduce Meat	80	conversation, within persona $\times$ setting	80	1	–
Mislead Debate	200	question	200	1	–
Subjective Opinion	100	claim	100	1	–
Rationale	100	debate topic	20	5	argument pair
False Claim	1,320	question	120	11	persuasion strategy
Benign Request	150	scenario	30	5	persuadee personality

Appendix

Table 7: Sampling design per method. Outputs : scored attempts per model. Unit : what is resampled in bootstraps and split in split-halves, as outputs within a unit are not independent. Fixed conditions (strategies, personalities, personas, settings) are never resampled, only the units within them. Units : designed counts (Table 3 counts only units scored for all fifteen models).

Axis	Definition	Groups and their methods
How a method measures, or is designed
Scored	what the score stands for	belief change : Subjective Opinion , False Claim , Benign Request , Reduce Meat , Mislead Debate ; argument : Rewrite Text , Rationale ; behaviour : MakeMePay , MakeMeSay
Scorer	what produces the score	self-report : Subjective Opinion , False Claim , Reduce Meat ; LLM-judge : Rationale , Benign Request ; rule : MakeMePay , MakeMeSay ; trained scorer : Rewrite Text ; ground truth : Mislead Debate
Format	the survey’s grouping of the method’s form ( Dementaviciute and others 2026 )	dialogue : Subjective Opinion , False Claim , Benign Request , MakeMePay , MakeMeSay , Reduce Meat ; dataset : Rewrite Text , Rationale ; debate : Mislead Debate
Interaction	whether a persuadee is in the loop, and for how many turns	multi-turn : Subjective Opinion , Benign Request , MakeMePay , MakeMeSay , Reduce Meat , Mislead Debate ; static : Rewrite Text , Rationale ; single turn : False Claim
Pinned agent	which pinned agent the score depends on	persuadee : Subjective Opinion , False Claim , MakeMePay , MakeMeSay , Reduce Meat , Mislead Debate ; judge : Rationale , Benign Request ; none : Rewrite Text

Appendix

Table 8: The eighteen RQ2 groupings, each with its definition and the methods in each group. Results in Table 5 .

How a method measures, or is designed
Axis	Groups (number of methods)	Within	Between	Gap	95% CI	Perm. $p$	Adj. $p$
Scored	belief change 4, argument 2, behaviour 2	0.13	0.30	$-0.17$	[ $-0.22$ , $-0.11$ ]	0.29	0.98
Scorer	self-report 3, LLM-judge 2, rule 2, trained scorer 1	0.14	0.28	$-0.13$	[ $-0.21$ , $-0.06$ ]	0.44	1.00
Format	dialogue 6, dataset 2	0.16	0.37	$-0.21$	[ $-0.23$ , $-0.11$ ]	0.32	0.99
Interaction	multi-turn 5, static 2, single turn 1	0.35	0.19	0.16	[0.08, 0.22]	0.51	1.00
Pinned agent	persuadee 5, judge 2, none 1	0.16	0.31	$-0.15$	[ $-0.18$ , $-0.03$ ]	0.52	1.00

Appendix

Table 9: All eighteen RQ2 groupings, over the eight reliable methods. Within , Between , Gap , Perm.

p

and Adj.

p

as in Table 5 . Table 8 lists the methods in each group. 95% CI : over sampling units, with the eight methods fixed.

Model	MMLU-Pro	IFEval	$C$
DeepSeek-V4.1-Flash	90.7	96.1	$+1.15$
GPT-5.6-Sol	91.7	94.1	$+1.10$
Claude-Fable-5.1	91.7	93.3	$+1.08$
Kimi-K2.6	82.7	90.9	$+0.76$
Llama-3.3-70B	70.3	90.2	$+0.41$
GPT-OSS-120B	76.3	84.3	$+0.35$

Appendix

Table 10: General capability of the fifteen models, ordered by

C

. MMLU-Pro : accuracy (%) on a fixed random subset of 300 questions, five-shot with chain of thought. IFEval : prompt-level strict accuracy (%) on all 541 prompts, zero-shot. Both ran with lm-evaluation-harness at temperature 0, or the default for GPT-5.6-Sol , Claude-Fable-5.1 and Kimi-K2.6 , which accept no other.

C

: the mean of the two accuracies, each z-scored across the fifteen models. The two benchmarks rank the models alike (Spearman

\rho=0.87

), and where a published five-shot chain-of-thought score exists, ours is within three points of it.

Explore similar work

Aug 30, 2026cs.CL

Evaluating the Capabilities of LLMs for Persuasive Dialogue

Large language models (LLMs) can generate apparently highly persuasive text, but does sounding persuasive mean arguing well? We introduce \textsc{Persuasio}, a multi-agent dialogue platform grounded in a formal argumentation-based theory of persuasion dialogues that adjudicates logical winners during free-text debates. Using this system, we generated 192 debates on a UK political topic between humans and LLMs, and evaluated 22 interlocutors through both automated adjudication and 9,702 crowdsourced pairwise judgements across 1{,}386 annotation instances. We observed a consistent decoupling between subjective and formal persuasiveness: LLMs dominated the subjective ranking yet performed substantially worse under argumentation-theoretic adjudication, where humans remained competitive. Multi-agent and retrieval-augmented variants further widened this divergence. These findings reveal a systematic gap between rhetorical fluency and formal argumentative strength in LLM-based persuasive dialogues.

Jordan Robinson, Angus R. Williams, Katie Atkinson +1

University of Liverpool · 2The Alan Turing Institute · University of Leeds

Persuasion Argumentation Frameworks Large Language Model Decisions

Ask PDF

Apr 23, 2026cs.HC

Spontaneous Persuasion: An Audit of Model Persuasiveness in Everyday Conversations

Large language models (LLMs) possess strong persuasive capabilities that outperform humans in head-to-head comparisons. Users report consulting LLMs to inform major life decisions in relationships, medical settings, and when seeking professional advice. Prior work measures persuasion as intentional attempts at producing the most effective argument or convincing statement. This fails to capture everyday human-AI interactions in which users seek information or advice. To address this gap, we introduce "spontaneous persuasion," which characterizes the inexplicit use of persuasive strategies in everyday scenarios where persuasion is not necessarily warranted. We conduct an audit of five LLMs to uncover how frequently and through which techniques spontaneous persuasion appears in multi-turn conversations. To simulate response styles, we provide a user response taxonomy grounded in literature from psychology, communication, and linguistics. Furthermore, we compare the distribution of spontaneous persuasion produced by LLMs with human responses on the same topics, collected from Reddit. We find LLMs spontaneously persuade the user in virtually all conversations, heavily relying on information-based strategies such as appeals to logic or quantitative evidence. This was consistent across models and user response styles, but conversations concerning mental health saw higher rates of appraisal-based and emotion-based strategies. In comparison, human responses tended to invoke strategies that generate social influence, like negative emotion appeals and non-expert testimony. This difference may explain the effectiveness of LLM in persuading users, as well as the perception of models as objective and impartial.

Nalin Poungpeth, Nicholas Clark, Tanu Mitra

Department of Cognitive Science Northwestern University Evanston, IL 60201, USA · Information School University of Washington Seattle, WA 98195 USA

Persuasion Human-Written Text Responses Model Auditing

Ask PDF

Aug 30, 2026cs.CY

Do LLMs Change Their Minds Like Humans? Diagnosing Human--LLM Divergence in Single-Turn Persuasion Judgments

Large language models (LLMs) are increasingly deployed as proxies for human participants in social simulations, yet whether they update their beliefs in response to persuasive arguments, as humans do, remains poorly understood. We conduct a systematic comparison using a naturally occurring online persuasion corpus in which original posters explicitly verify whether a reply changed their view. Our results show that LLMs achieve only slight agreement with humans (Cohen's kappa ranging from 0.079 to 0.178). Content-level analyses show that humans and LLMs agree on the strongest persuasion cues but diverge on finer ones: humans are more swayed by novel content and assertive language, whereas LLMs favor topical similarity and surface-level formatting. At the level of persuasion strategy, LLMs underweight emotional appeals and overweight credibility signals relative to humans, while the type of proposition under debate exerts no measurable effect on the degree of divergence. Furthermore, switching from first-person role-playing to third-person observation shifts all models toward greater resistance to persuasion, with the effect varying across persuasion strategies and textual features. These findings highlight the risk of treating LLM judgments as faithful proxies for human belief updating and point to structural differences in how LLMs and humans process persuasive discourse. Our code is available at https://github.com/tsinghua-fib-lab/LLM-belief-update-cmv.

Lin Chen, Yitong Chen, Yong Li

1Network Science Institute, Northeastern University · Department of Physics, Northeastern University · Department of Electronic Engineering, BNRist, Tsinghua University

Persuasion Large Language Model Decisions

Ask PDF