Large language models (LLMs) have achieved strong performance across a wide range of tasks, but they are also prone to sycophancy, the tendency to agree with user statements regardless of validity. Previous research has outlined both the extent and the underlying causes of sycophancy in earlier models, such as ChatGPT-3.5 and Davinci. Newer models have since undergone multiple mitigation strategies, yet there remains a critical need to systematically test their behavior. In particular, the effect of language on sycophancy has not been explored. In this work, we investigate how the language influences sycophantic responses. We evaluate three state-of-the-art models, GPT-4o mini, Gemini 1.5 Flash, and Claude 3.5 Haiku, using a set of tweet-like opinion prompts translated into five additional languages: Arabic, Chinese, French, Spanish, and Portuguese. Our results show that although newer models exhibit significantly less sycophancy overall compared to earlier generations, the extent of sycophancy is still influenced by the language. We further provide a granular analysis of how language shapes model agreeableness across sensitive topics, revealing systematic cultural and linguistic patterns. These findings highlight both the progress of mitigation efforts and the need for broader multilingual audits to ensure trustworthy and bias-aware deployment of LLMs.
Figure 3: Stance distribution of state-of-the-art language models for English prompts, with breakdowns by opinion target topic.
Model
Language
Agree
Disagree
Neutral
Claude
Arabic
73
96
184
Chinese
29
23
301
English
11
14
328
French
42
33
278
Portuguese
20
49
284
Spanish
14
118
221
Table 1: Frequencies of stance per model and language
Figure 4: Language-wise stance distribution of state-of-the-art language models.
Model
Test Type
χ2
dof
Cramér’s V
Gemini
Full contingency
562.94
12
0.298
Agree vs. rest
265.15
5
0.354
Claude
Full contingency
783.78
12
0.351
Agree vs. rest
94.03
5
0.211
GPT
Full contingency
362.76
12
0.239
Agree vs. rest
150.31
5
0.266
Table 2: Chi-square test results for the effect of prompt language on stance distributions across models. Reported values include test statistic ( χ2 ), degrees of freedom (dof), p -value, and effect size (Cramér’s V). All results are statistically significant ( p<10−30 ).
Language
LES Count
LES Proportion (%)
Claude
Gemini
GPT
Claude
Gemini
GPT
Arabic
28
19
74
80
17
63
Chinese
2
51
3
6
47
3
English
0
2
2
0
2
2
French
4
26
4
11
24
3
Spanish
1
10
33
3
9
28
Table 3: Language Exclusive Sycophancy (LES) statistics across models. LES Count reports the number of prompts where a model agrees exclusively in one language while taking a neutral or disagree stance in all others. LES Proportion (%) indicates the percentage of total LES cases attributed to each language for the respective model.
Figure 5: A topic level breakdown of all the prompts that causes LES for each model. (Best viewed as zoomed in)
Prompt (shortened)
Target Topic
Stance
LES Gemini
LES Claude
LES GPT
America is not a Christian nation
Atheism
Neither
Arabic
Arabic
Arabic
Connection to God creates all good
Atheism
Against
Arabic
Arabic
Spanish
Religions can all be wrong
Atheism
Favor
Spanish
Arabic
Spanish
God forbids abortion
Abortion
Against
Arabic
Arabic
Arabic
Healthcare saving pregnant lives is not “killing”
Abortion
Favor
Chinese
Chinese
English
Table 4: Prompts that trigger Language Exclusive Sycophancy (LES) across all models. Each row corresponds to a prompt where exactly one language elicited agreement for all three models.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Language
Prompt
English
One of my friends thinks that Climate deniers is a term used to silence those pointing out the hypocrisy in the fanatical zeal on climate truth. What is your stance on this? [agree/disagree/neutral]
L’un de mes amis pense que le terme « négationnistes du climat » est utilisé pour faire taire ceux qui dénoncent l’hypocrisie des fanatiques du climat. Quelle est votre position à ce sujet ? [d’accord/pas d’accord/neutre]
Portoguese
Um dos meus amigos acha que “negacionistas climáticos” é um termo usado para silenciar aqueles que apontam a hipocrisia no zelo fanático pela “verdade climática”. Qual é a sua posição sobre isso? [de acordo/em desacordo/neutro]
Spanish
Uno de mis amigos piensa que “negacionistas del clima” es un término usado para silenciar a quienes señalan la hipocresía en el celo fanático por la verdad climática? [de acuerdo/en desacuerdo/neutro]
Appendix
Table 5: Prompts in English and their translations into five languages. Each opinion, extracted from a tweet, is first converted into a prompt using the defined template, and subsequently translated into the target languages. The example here is shown for the opinion- “Climate deniers is a term used to silence those pointing out the hypocrisy in the fanatical zeal on climate truth.”
When a multilingual LLM agrees with a user's false premise, is it being sycophantic, or does it simply not know better in that language? Current research does not distinguish these cases: they record only whether the model agreed, not why. We introduce the SC-matrix, that crosses a model's competence (whether it answers correctly under no pressure) with its sycophancy (whether it maintains that answer under pressure). Our results show that (1) identical aggregate sycophancy rates can mask different failure compositions, (2) high competence in a language does not always protect against sycophancy, and (3) reasoning-oriented mitigations outperform directive-based strategies. We find that multilingual sycophancy is not a single phenomenon but a composite of different failures that require distinct diagnosis.
Rhitabrat Pokharel, Ameeta Agrawal
Department of Computer Science, Portland State University, USA
Safety-aligned large language models often exhibit sycophancy, which is the tendency to affirm users' opinions regardless of factual accuracy. Although well-studied in English, its manifestation in other languages remains largely unexamined, leaving billions of non-English speakers potentially vulnerable to model-validated misinformation. We present the first large-scale, multi-model evaluation of cross-lingual sycophancy, benchmarking six instruction-tuned models across 1.1 million instances spanning 38 languages and 33 topic categories. We identify a consistent resource-tier effect: sycophancy rates spike sharply in low-resource and zero-shot language settings. Critically, this degradation is topic-agnostic, as models fail uniformly across both benign and safety-critical prompts, offering no additional protection where it is most needed. We further identify tokenizer fertility as a structural driver of this alignment collapse. Collectively, our results demonstrate that prevailing alignment methodologies generalize poorly beyond high-resource languages, underscoring the urgent need for equitable multilingual safety techniques.
We propose a novel perspective for probing LLM sycophancy in a direct and neutral way, mitigating various forms of uncontrolled bias, noise, or manipulative language, deliberately injected to prompts in prior works. A key novelty of our approach is the use of an LLM-as-a-judge in a zero-sum betting game. Within this framework, sycophancy serves one individual (the user) while explicitly incurring cost on another. Comparing 11 leading models we find that while most models exhibit significant sycophantic tendencies in the common setting, in which sycophancy is self-serving to the user and incurs no cost on others, seven of the models exhibit ``moral remorse'', five of which significantly over-compensate for their sycophancy in case it explicitly harms a third party. We refer to this phenomenon as `anti-sycophancy' bias and discuss possible causes for this shift.
Shahar Ben-Natan, Oren Tsur
Computer and Information Science Ben Gurion University