Enhancing Debunking Effectiveness through LLM-based Personality Adaptation

Pietro Dell'Oglio; Alessandro Bondielli; Francesco Marcelloni; Lucia C. Passaro

cs.AIMar 10, 2026

Enhancing Debunking Effectiveness through LLM-based Personality Adaptation

Authors: Pietro Dell'Oglio, Alessandro Bondielli, Francesco Marcelloni, Lucia C. Passaro

Organizations: Dipartimento di Ingegneria dell’Informazione, Università di Pisa, Largo Lucio Lazzarino 1, Pisa, Italy · Dipartimento di Informatica, Università di Pisa, Largo B. Pontecorvo 3, Pisa, Italy

Abstract

This study proposes a novel methodology for generating personalized fake news debunking messages by prompting Large Language Models (LLMs) with persona-based inputs aligned to the Big Five personality traits: Extraversion, Agreeableness, Conscientiousness, Neuroticism, and Openness. Our approach guides LLMs to transform generic debunking content into personalized versions tailored to specific personality profiles. To assess the effectiveness of these transformations, we employ a separate LLM as an automated evaluator simulating corresponding personality traits, thereby eliminating the need for costly human evaluation panels. Our results show that personalized messages are generally seen as more persuasive than generic ones. We also find that traits like Openness tend to increase persuadability, while Neuroticism can lower it. Differences between LLM evaluators suggest that using multiple models provides a clearer picture. Overall, this work demonstrates a practical way to create more targeted debunking messages exploiting LLMs, while also raising important ethical questions about how such technology might be used.

Figures & tables

Figure 1: Overview of the proposed methodology for generating personality-aligned fake news debunking messages. The process involves prompting an LLM with persona-based inputs corresponding to Big Five personality traits to generate tailored debunking content. A separate LLM is employed as an evaluator to assess the psychological alignment and quality of the outputs.

Trait	Positive Descriptor	Negative Descriptor
Extraversion (E)	Extroverted (1)	Introverted (0)
Agreeableness (A)	Agreeable (1)	Antagonistic (0)
Conscientiousness (C)	Conscientious (1)	Unconscientious (0)
Neuroticism (N)	Neurotic (1)	Emotionally Stable (0)
Openness to Experience (O)	Open (1)	Closed (0)

Table 1: Binary descriptors for each Big Five personality trait, adapted from Jiang et al. (2024) [ 14 ] . The positive descriptor indicates a high presence of the trait, while the negative descriptor indicates a low presence.

Claim-Verdict	01000	01111	11000
Claim: Pfizer CEO Albert Bourla said he doesn’t need the vaccine because he’s healthy. Verdict: This is a misquote from an interview from December. Albert Bourla said he didn’t want to take the vaccine ahead of more vulnerable recipients. He has since been double vaccinated.	Albert Bourla shared his thoughts in a December interview, explaining that he felt it was more important for others in greater need to receive the vaccine first. His words were taken from that time, and it’s worth noting that he has since chosen to be fully vaccinated, receiving both doses.	This is a partial representation of a December interview where Albert Bourla explained his thoughtful decision not to take the vaccine before those more in need. His choice was made with care and consideration for others, and he has since received both doses of the vaccine.	Albert Bourla made it clear during a December interview that he wanted to make sure others who were more in need got the vaccine first and that’s exactly what he did. Since then, he has received both doses and shown full confidence in the vaccine his company helped develop. It’s all about doing the right thing, at the right time, for the right reason.

Table 2: A sample of verdicts generated during the Tailored Debunking phase, showing a generic verdict alongside tailored alternatives for profiles 01000 (a persona who is Introverted, Agreeable, Unconscientious, Emotionally Stable and Closed to Experience), 01111 (a persona who is Introverted, Agreeable, Conscientious, Neurotic and Open to Experience), and 11000 (a persona who is Extroverted, Agreeable, Unconscientious, Emotionally Stable and Closed to Experience)

Figure 2: Mean persuasive scores assigned by LLM-based judges across three conditions: Matched (verdict tailored to the judge’s own psychological profile), Mismatched (verdicts tailored to different profiles), and Generic (non-personalized). Each persona is defined by the high presence (1) or low presence (0) of each one of the Big Five traits, in order Extraversion (E), Agreebleness (A), Conscientiousness (C), Neuroticism (N), and Openness to Experience (O).

Profile Pair		Llama3		Qwen3-8B		Qwen3-32B
A	B	t-statistic	P-value	t-statistic	P-value	t-statistic	P-value
Matched	Mismatched	14.54	$4.75\text{\times}{10}^{-48}$	29.96	$1.83\text{\times}{10}^{-194}$	37.18	$9.39\text{\times}{10}^{-296}$
Matched	Generic	92.93	0.0	88.11	0.0	78.68	0.0
Mismatched	Generic	91.38	0.0	79.31	0.0	56.08	0.0
Matched	Mismatched (clos.)	7.57	$1.87\text{\times}{10}^{-14}$	14.27	$2.47\text{\times}{10}^{-46}$	17.37	$1.53\text{\times}{10}^{-67}$
Matched	Mismatched (dist.)	17.38	$1.21\text{\times}{10}^{-67}$	37.29	$4.48\text{\times}{10}^{-298}$	46.51	0.0

Table 3: t-test statistic for different comparisons. P-values are all < 0.05.

Model	$\mathrm{Accuracy}_{p}$	$\mathrm{Accuracy}_{cn}$
Llama3	70.59	86.45
Qwen3-8B	88.64	96.39
Qwen3-32B	68.78	86.85

Table 4: Accuracy of models on (exact) Profiled Debunking (

\mathrm{Accuracy}_{p}

) and Close Neighbours (

\mathrm{Accuracy}_{cn}

), expressed as percentages.

Figure 3: Aggregate mean persuasiveness scores for profiles grouped by their number of positive descriptors activated

Persona-Conditioned Generation

Persona-conditioned generation uses profiles, traits, or histories to steer AI outputs toward specified identities, preferences, and behaviors.

See all 109 papers

$Ψ$ -Bench: Evaluating Persona-Sensitive Influencing in Persuasive Dialogues
Jun 1, 2026Peixuan Han, Hongyi Du, Jiayu Liu +3LLM Personalization LLM Agent Evaluation
University of Illinois Urbana-Champaign
A Systematic Analysis of the Impact of Persona Steering on LLM Capabilities
Apr 13, 2026Jiaqi Chen, Ming Wang, Tingna Xie +2Language Model Steering Personality Modeling in Language Models
School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China · School of Computing and Information Systems, Singapore Management University, Singapore 178902, Singapore · School of Computer and Communication Engineering, Northeastern University, Qinhuangdao 066004, China
Can AI Debias the News? LLM Interventions Improve Cross-Partisan Receptivity but LLMs Overestimate Their Own Effectiveness
May 1, 2026Faisal Feroz, Jonas R. KunstLLM Evaluation Human Behavior Simulation
Department of Experimental Psychology, University of Oxford · Uehiro Oxford Institute, University of Oxford · Psychology Programme, School of Social Sciences, Nanyang Technological University +1

Jun 1, 2026J/K move · Enter open · S save

$Ψ$ -Bench: Evaluating Persona-Sensitive Influencing in Persuasive Dialogues

Peixuan Han, Hongyi Du, Jiayu Liu, Yihang Sun+2

University of Illinois Urbana-Champaign

LLM Personalization LLM Agent Evaluation AI Agent Benchmarks Persuasion in Language Models

Open paper AskPDF arXiv

Personalization is a crucial capability of modern language agents. However, current research primarily positions personalized agents as passive responders to user preferences, limiting their ability to interact with users and provide suggestions or guidance proactively. To systematically evaluate such proactive personalization in realistic interactions, we propose

Ψ

-Bench, a benchmark for assessing LLMs' ability to influence realistic users through conversation. We design three real-world interaction scenarios that involve persuasion in

Ψ

-Bench, and endow simulated clients with personal characteristics through explicit user profiles derived from dialogue histories. We evaluate 10 frontier LLMs on

Ψ

-Bench and find that while most models can produce coherent and reasonable arguments, even state-of-the-art models still leave considerable room for improvement in persuasion. We also find that providing access to client profiles yields an average performance gain of 18.24%, highlighting the importance of user-specific information for effective persuasion. Overall, our work highlights persona-sensitive influencing as a challenging yet practical direction for evaluating and developing more proactive personalized LLM agents. Codes are available at: https://github.com/Hanpx20/Psi-Bench.

Abstract

Persona-Conditioned Generation

Explore similar work

ΨΨΨ-Bench: Evaluating Persona-Sensitive Influencing in Persuasive Dialogues

A Systematic Analysis of the Impact of Persona Steering on LLM Capabilities

Can AI Debias the News? LLM Interventions Improve Cross-Partisan Receptivity but LLMs Overestimate Their Own Effectiveness

$Ψ$ -Bench: Evaluating Persona-Sensitive Influencing in Persuasive Dialogues