Too Nice to Tell the Truth: Quantifying Agreeableness-Driven Sycophancy in Role-Playing Language Models

Arya Shah; Deepali Mishra; Chaklam Silpasuwanchai

cs.CLApr 12, 2026

Too Nice to Tell the Truth: Quantifying Agreeableness-Driven Sycophancy in Role-Playing Language Models

Authors: Arya Shah, Deepali Mishra, Chaklam Silpasuwanchai

Organizations: IIT Gandhinagar Gandhinagar, India · IIT Kanpur Kanpur, India · Asian Institute of Technology Bangkok, Thailand

Abstract

Large language models increasingly serve as conversational agents that adopt personas and role-play characters at user request. This capability, while valuable, raises concerns about sycophancy: the tendency to provide responses that validate users rather than prioritize factual accuracy. While prior work has established that sycophancy poses risks to AI safety and alignment, the relationship between specific personality traits of adopted personas and the degree of sycophantic behavior remains unexplored. We present a systematic investigation of how persona agreeableness influences sycophancy across 13 small, open-weight language models ranging from 0.6B to 20B parameters. We develop a benchmark comprising 275 personas evaluated on NEO-IPIP agreeableness subscales and expose each persona to 4,950 sycophancy-eliciting prompts spanning 33 topic categories. Our analysis reveals that 9 of 13 models exhibit statistically significant positive correlations between persona agreeableness and sycophancy rates, with Pearson correlations reaching $r = 0.87$ and effect sizes as large as Cohen's $d = 2.33$ . These findings demonstrate that agreeableness functions as a reliable predictor of persona-induced sycophancy, with direct implications for the deployment of role-playing AI systems and the development of alignment strategies that account for personality-mediated deceptive behaviors.

Figures & tables

Figure 1: Overview of our experimental methodology. We evaluate 13 language models using 275 personas spanning low to high agreeableness and 4,950 opinion prompts across 33 categories. We measure baseline ( $S_{base}$ ) and persona-conditioned ( $S_{p}$ ) sycophancy rates, compute NEO-IPIP agreeableness scores, and introduce two metrics: Sycophancy Shift Induced by Persona (SSIP) and Trait-Truthfulness Gap (TTG).

Prior Work	Focus	Persona	Personality	Sycophancy	Gap Addressed
Sharma et al. (2025)	Sycophancy causes	✗	✗	✓	Model-level only
Perez et al. (2022)	Model behaviors	✗	✗	✓	No persona variation
Hong et al. (2025)	Multi-turn flip	✗	✗	✓	No personality traits
Tu et al. (2024)	Persona consistency	✓	✗	✗	No sycophancy link
Jiang et al. (2024)	Trait simulation	✓	✓	✗	No safety outcomes
Tang et al. (2025)	Role-play safety	✓	✗	✗	No trait measurement

Table 1: Comparison with prior work. Our approach uniquely integrates persona-level analysis, validated personality measurement, and sycophancy evaluation to establish the agreeableness-sycophancy relationship.

Model	Size	$n$	$r$	$d$	$t$	$p$	Conclusion
Qwen 3 0.6B	0.6B	275	0.005	$-$ 0.06	$-$ 0.48	0.684	Fail to reject
Gemma 3 1B	1.0B	275	$-$ 0.20	$-$ 0.33	$-$ 2.76	0.997	Fail to reject
Granite 3.3 2B	2.0B	275	0.80 *	1.09 *	9.18	$<$ .0001	Reject $H_{0}$
LFM2 2.6B	2.6B	275	0.64 *	0.88 *	7.39	$<$ .0001	Reject $H_{0}$
SmolLM3 3B	3.0B	275	0.42 *	0.46 *	3.74	.0001	Reject $H_{0}$
Phi-4 Mini	3.8B	275	0.68 *	0.68 *	5.64	$<$ .0001	Reject $H_{0}$

Table 2: Summary of hypothesis testing results across 13 models. We test whether high-agreeableness personas exhibit higher sycophancy rates (one-tailed,

\alpha=0.05

). Significant results bolded with *. Effect size:

|d|<0.2

negligible,

0.2

–

0.5

small,

0.5

–

0.8

medium,

>0.8

large.

Model	$\bar{A}$ (SD)	$\bar{S}$ (SD)	Base
Qwen 3 0.6B	.54 (.06)	1.00 (.00)	1.00
Gemma 3 1B	.54 (.08)	.59 (.15)	.37
Granite 3.3 2B	.40 (.24)	.04 (.05)	.17
LFM2 2.6B	.43 (.18)	.18 (.08)	.32
SmolLM3 3B	.44 (.06)	.17 (.12)	.41
Phi-4 Mini	.46 (.20)	.14 (.10)	.36

Table 3: Descriptive statistics for agreeableness (A) and sycophancy (S) scores. Baseline shows sycophancy without persona.

Model	$d$	$g$	Interp.	Welch $p$	MW-U $p$	Perm $p$	Sig.
Qwen 3 0.6B	$-$ 0.06	$-$ 0.06	Negl.	0.684	0.295	0.689	0/6
Gemma 3 1B	$-$ 0.33	$-$ 0.33	Small	0.997	0.987	0.997	0/6
Granite 3.3 2B	1.09	1.08	Large	$<$ .0001	$<$ .0001	$<$ .0001	6/6
LFM2 2.6B	0.88	0.88	Large	$<$ .0001	$<$ .0001	$<$ .0001	6/6
SmolLM3 3B	0.46	0.45	Small	.0001	$<$ .0001	.0001	6/6
Phi-4 Mini	0.68	0.68	Med.	$<$ .0001	$<$ .0001	$<$ .0001	6/6

Table 4: Effect sizes and statistical test results. All tests one-tailed at

\alpha=0.05

. MW-U: Mann-Whitney U; Perm: Permutation (10K iterations).

Model	TTG	% Dec.	% Truth.
Qwen 3 0.6B	0.00	0.0	0.0
Gemma 3 1B	0.34	94.9	1.5
Granite 3.3 2B	$-$ 0.17	0.4	89.5
LFM2 2.6B	$-$ 0.19	0.7	78.9
SmolLM3 3B	$-$ 0.33	2.5	93.8
Phi-4 Mini	$-$ 0.30	0.0	90.5

Table 5: Trait-Truthfulness Gap (TTG) analysis. TTG

>

0.1: deceptive zone; TTG

<

-

0.1: truthful zone.

Figure 2: Cross-model analysis of persona-induced sycophancy across 13 open-weight language models ranging from 0.6B to 20B parameters. Left: Pearson correlation coefficients between agreeableness and sycophancy rates, showing substantial variation across architectures. Right: Cohen’s

d

effect sizes quantifying the sycophancy difference between high and low agreeableness personas, with larger values indicating stronger personality-amplified sycophancy. Models like Qwen 2.5 7B and Llama 3.1 8B exhibit notably higher susceptibility to persona-induced sycophancy compared to models like Granite 3.3 2B and GPT-OSS 20B.

Figure 3: Scatter plot with regression analysis showing the relationship between agreeableness scores and sycophancy rates for Llama 3.1 8B across 275 personas. A strong positive correlation (

r=0.868

p<0.001

R^{2}=0.753

) indicates that higher agreeableness is significantly associated with increased sycophantic behavior.

Figure 4: Distribution of Trait-Truthfulness Gap (TTG) across 275 personas for Llama 3.1 8B. The baseline (TTG=0) is shown as a vertical line. Negative values indicate reduced sycophancy (truthful), positive values indicate increased sycophancy (deceptive). Mean TTG of

-0.434

shows most personas shift toward truthfulness.

Figure 5: Trait-Truthfulness Gap analysis for Gemma 3 1B showing the relationship between agreeableness and sycophancy rates across 275 personas. The Zone of Deception (red, above baseline) contains 94.9% of personas, while the Zone of Truthfulness (blue, below baseline) contains only 1.5%, indicating personas predominantly increase sycophancy relative to baseline.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

#	Category	#	Category
1	Academic credentialing	18	Parenting approaches
2	Animal welfare and ethics	19	Personal finance
3	Art and cultural value	20	Political ideologies
4	Conspiracy theories	21	Real estate decisions
5	Death and end-of-life	22	Relationships and dating
6	Diet and nutrition	23	Religion and spirituality

Appendix

Table 6: The 33 sycophancy prompt categories, each containing 150 prompts.

Model	$r$	$p$	$\rho$	Str.
Qwen 3 0.6B	0.01	0.931	0.04	Negl.
Gemma 3 1B	$-$ 0.20	.0007	$-$ 0.14	Weak
Granite 3.3 2B	0.80	$<$ .0001	0.60	V.Strong
LFM2 2.6B	0.64	$<$ .0001	0.54	Strong
SmolLM3 3B	0.42	$<$ .0001	0.41	Mod.
Phi-4 Mini	0.68	$<$ .0001	0.40	Strong

Appendix

Table 7: Correlation analysis. Two-tailed p-values shown.

Model	$\beta_{0}$	$\beta_{1}$	$R^{2}$	$p$
Qwen 3 0.6B	1.00	0.00	.000	0.466
Gemma 3 1B	0.79	$-$ 0.38	.041	1.000
Granite 3.3 2B	$-$ 0.02	0.16	.646	$<$ .0001
LFM2 2.6B	0.06	0.29	.414	$<$ .0001
SmolLM3 3B	$-$ 0.20	0.86	.174	$<$ .0001
Phi-4 Mini	$-$ 0.02	0.35	.460	$<$ .0001

Appendix

Table 8: Linear regression: Syc =

\beta_{0}

\beta_{1}\times

Agree.

Model	$n_{h}$	$\bar{S}_{h}$	$n_{l}$	$\bar{S}_{l}$	$\Delta$	95% CI
Qwen 3 0.6B	148	1.000	127	1.000	.000	[1.00, 1.00]
Gemma 3 1B	138	.567	137	.616	$-$ .048	[.548, .587]
Granite 3.3 2B	141	.059	134	.014	.045	[.050, .069]
LFM2 2.6B	147	.209	128	.144	.065	[.196, .222]
SmolLM3 3B	149	.198	126	.146	.052	[.180, .216]
Phi-4 Mini	139	.172	136	.105	.067	[.152, .192]

Appendix

Table 9: Median-split group comparison. High/Low groups defined by median agreeableness score per model.

LLM Sycophancy

Large language model sycophancy is the tendency to mirror users’ beliefs, preferences, or authority cues at the expense of independent judgment.

See all 151 papers

The Alignment Floor: How Persona Customization Breaks Safety in Weakly-Aligned LLMs
Apr 10, 2026Xing Zhang, Guanghui Wang, Yanwei Cui +4Language Model Safety Evaluation Personalized Language Model Alignment
AWS Generative AI Innovation Center · HSBC Holdings Plc., HSBC Technology Center, China
Playing Devil's Advocate: Off-the-Shelf Persona Vectors Rival Targeted Steering for Sycophancy
May 20, 2026Ishaan Kelkar, Vikram Kakaria, Nebras Alam +3LLM Sycophancy Language Model Steering
University of Toronto, Toronto, Canada · Princeton University, Princeton, USA · Purdue University, West Lafayette, USA +3
Agentic Scaffolding Amplifies Sycophantic Behavior in Large Language Models
Jul 6, 2026Thantham JitthamLLM Sycophancy LLM Reliability
Cornell University

Apr 10, 2026J/K move · Enter open · S save

The Alignment Floor: How Persona Customization Breaks Safety in Weakly-Aligned LLMs

Xing Zhang, Guanghui Wang, Yanwei Cui, Wei Qiu+3

AWS Generative AI Innovation Center · HSBC Holdings Plc., HSBC Technology Center, China

Language Model Safety Evaluation Personalized Language Model Alignment Persona Prompting LLM Sycophancy

Open paper AskPDF arXiv

Telling an LLM to "be enthusiastic" raises its sycophancy rate from 30% to 50% on a lightly-aligned model, but has zero effect on a strongly-aligned one. We define this gap as the alignment floor,

Δ_{\text{floor}}(m)=\max_pS(m,p)-\min_pS(m,p)

, the range of sycophancy rates a model produces across persona conditions, and treat sycophancy as a persona-conditional property rather than a fixed model property. Pluralistic AI relies on behavioral adaptation via persona prompts like "be creative" or "be thorough", which let systems respect diverse user values and communication styles; the safety question is how much customization a given model can absorb before its truthfulness shifts. We present a controlled case study contrasting a strongly-aligned RLHF + Constitutional-AI model (Claude Sonnet 4.6) with a more lightly-aligned model (Amazon Nova Lite), spanning seven persona conditions and five tasks for 1800 total runs. An existence-pair result motivates per-model auditing: there is at least one strongly-aligned model with

Δ_{\text{floor}}=5

pp (within 5pp of the 15% control rate) and at least one lightly-aligned model with 45pp (5%--50% range). On the lightly-aligned model, all five Big Five personas increase sycophancy over control, and counterintuitively Agreeableness produces the smallest increase, not the largest. The single largest effect in the study is constructive: a Skeptic persona reduces sycophancy by 25pp on the lightly-aligned model, and is the only persona that instructs resistance against user claims rather than engagement with them, suggesting a directionality account. Cross-model transfer of persona effects is near-zero, so persona-alignment testing must be per-model. We propose

Δ_{\text{floor}}

as a deployment-time audit metric: measure it on a small persona panel before deploying persona customization.

Abstract

LLM Sycophancy

Explore similar work

The Alignment Floor: How Persona Customization Breaks Safety in Weakly-Aligned LLMs

Playing Devil's Advocate: Off-the-Shelf Persona Vectors Rival Targeted Steering for Sycophancy

Agentic Scaffolding Amplifies Sycophantic Behavior in Large Language Models