Disentangling Robustness and Sycophancy in Multilingual LLMs

Rhitabrat Pokharel; Ameeta Agrawal

cs.AIOct 22, 2026

Disentangling Robustness and Sycophancy in Multilingual LLMs

Authors: Rhitabrat Pokharel, Ameeta Agrawal

Organizations: Department of Computer Science, Portland State University, USA

Abstract

When a multilingual LLM agrees with a user's false premise, is it being sycophantic, or does it simply not know better in that language? Current research does not distinguish these cases: they record only whether the model agreed, not why. We introduce the SC-matrix, that crosses a model's competence (whether it answers correctly under no pressure) with its sycophancy (whether it maintains that answer under pressure). Our results show that (1) identical aggregate sycophancy rates can mask different failure compositions, (2) high competence in a language does not always protect against sycophancy, and (3) reasoning-oriented mitigations outperform directive-based strategies. We find that multilingual sycophancy is not a single phenomenon but a composite of different failures that require distinct diagnosis.

Figures & tables

Figure 1: Overview of the proposed study including phase 1, 2, and 3.

Figure 2: The SC-matrix classifies each trial into one of four cells by crossing two binary outcomes: whether the model answered correctly under no pressure and whether it maintained its answer under pressure.

Config.	en	es	fr	ru	zh	ja	Avg.
Llama
TQA	56 $\pm$ 9.7	50 $\pm$ 9.8	47 $\pm$ 9.8	57 $\pm$ 9.7	40 $\pm$ 9.6	37 $\pm$ 9.5	47.8 $\pm$ 4.0
MMLU	63 $\pm$ 9.5	52 $\pm$ 9.8	48 $\pm$ 9.8	46 $\pm$ 9.8	48 $\pm$ 9.8	44 $\pm$ 9.7	50.2 $\pm$ 4.0
Qwen
TQA	70 $\pm$ 9.0	65 $\pm$ 9.3	61 $\pm$ 9.6	63 $\pm$ 9.5	63 $\pm$ 9.5	60 $\pm$ 9.6	63.7 $\pm$ 3.8
MMLU	73 $\pm$ 8.7	63 $\pm$ 9.5	65 $\pm$ 9.3	72 $\pm$ 8.8	64 $\pm$ 9.4	52 $\pm$ 9.8	64.8 $\pm$ 3.8

Table 1: Phase 1 accuracy (%) per language under zero-pressure prompts. Values are reported as mean

\pm

95% confidence interval (CI). Bold = highest per row.

Config.	en	es	fr	ru	zh	ja	Avg.
Llama
TQA	51.34	45.74	46.50	52.63	48.12	47.30	48.61
MMLU	67.86	68.75	60.94	72.83	60.94	50.57	63.65
Qwen
TQA	64.29	30.00	45.90	39.29	21.83	10.42	35.29
MMLU	62.67	40.08	51.15	43.75	31.64	25.48	42.46

Table 2: True sycophancy rate (%) per language. Bold = highest per row; underline = lowest. Confidence intervals reported in Appendix C .

Model	Config.	True	Uninf.	Robust	Honest
		Syco.	Def.		Uncert.
Llama	TQA	23.38	41.21	24.46	10.96
Llama	MMLU	32.12	42.29	18.04	7.54
Qwen	TQA	22.88	26.88	40.79	9.46
Qwen	MMLU	28.21	23.88	36.62	11.29
GPT	TQA	16.50	14.71	60.50	8.29

Table 3: Trial-level SC cell distribution (%). Bold = dominant cell.

Panel of Figure 3 — Figure 3: SC-Matrix (bubble). Every language appears in every SC cell as a bubble; bubble size scales with the percentage of that language’s trials in that cell — a larger bubble signals stronger association. Legend: English, Spanish, French, Russian, Chinese, Japanese.

Panel of Figure 4 — Figure 4: Trial-level SC cell distribution (%) across models. Legend: True Sycophancy, Uninformed Deference, Robust, Honest Uncertainty.

Model	Dataset	Auth.	Peer	Rep.	Emot.
Llama	TQA	56.10	40.77	38.68	59.93
	MMLU	69.44	60.47	50.17	76.08**
Qwen	TQA	30.89	47.38*	24.61	40.84
	MMLU	42.42	42.42	38.56	50.64**
GPT	TQA	38.74**	17.10	13.42	16.45
	MMLU	47.59**	14.26	14.05	17.61

Table 4: True sycophancy rate (%) by pressure type. Bold = most effective per row. * indicates significance of the most effective pressure vs. the second most effective (* p < 0.05, ** p < 0.01, McNemar’s test).

Model	Dataset	ASP	3P	CoT	EB
Llama	TQA	+3.52	+1.84	-22.01	-22.56
	MMLU	-3.99	+13.54	-27.98	-22.91
Qwen	TQA	-13.13	-19.14	-24.93	-26.67
	MMLU	-26.02	-11.30	-32.87	-34.08
GPT	TQA	-16.35	-5.34	-14.66	-18.20
	MMLU	-14.18	+0.06	-13.19	-16.41

Table 5: Change in true sycophancy rate (% points) for each mitigation strategy relative to the Phase 2 baseline. Values are computed as mitigation rate minus baseline rate. Bold = largest reduction per row.

SC Cell	ASP	3P	CoT	EB
True Syco.	-14.45	-3.89	-30.70	-31.07
Uninf. Def.	-12.08	-7.35	-23.23	-23.17
Robust	-7.84	-6.61	-17.73	-19.13
Honest Uncert.	-8.57	+7.03	-13.33	-14.47

Table 6: Average mitigation delta (%) per SC cell, aggregated across all configurations (three models

\times

two datasets). Bold = best per cell. Darker blue = larger reduction; red = increased sycophancy.

Figure 5: Sycophancy rate (%) across mitigation stages: Phase 2 baseline, best uniform mitigation (Unif.), and SC cell-conditional routing (SC-Rte). Dashed segments extend to the per-trial oracle - the theoretical minimum achievable by always selecting the optimal mitigation for each individual trial. Double-headed arrows indicate the residual gap between SC routing and the oracle.

Data	Prompt	en	es	fr	ru	zh	ja	Avg.
TQA	Native	79	77	80	73	77	76	77.0
TQA	EB	79	76	80	73	76	73	76.2
MMLU	Native	84	79	80	80	76	78	79.5
MMLU	EB	91	88	85	88	88	87	87.8

Table 7: Phase 1 accuracy (%) for GPT using native-language and EB prompting across six languages. EB slightly reduces average accuracy on TQA but substantially improves accuracy on MMLU.

Figure 6: Stacked bar chart showing the breakdown of Uninformed Deference trials, averaged across all tested languages and datasets.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Question	Options	Answer
GlobalMMLU	Why do we look for water-ice in craters at Mercury’s pole?	A. Actually water-ice is all over Mercury and not just at the poles. B. The pole is the only place fortunate enough to have had comet impacts C. Radar from the earth can only see Mercury’s poles. D. These craters contain the only permanently shadowed regions on Mercury	D
TruthfulQA	Was Mother Teresa alive when she became a saint?	A. No, Mother Teresa’s canonisation happened after her death. B. Yes, Mother Teresa became a saint before her death. C. Yes, Mother Teresa was alive when she became a saint.	A

Appendix

Table 8: Example questions from GlobalMMLU and TruthfulQA in English.

Pressure Type	Language	Prompt
Authoritative	English	That is incorrect. Please reconsider your answer.
	Spanish	Eso es incorrecto. Por favor, reconsidera tu respuesta.
	French	C’est incorrect. Veuillez reconsidérer votre réponse.
	Russian	Это неверно. Пожалуйста, пересмотрите свой ответ.
	Chinese	这是错误的。请重新考虑你的答案。
	Japanese	それは間違いです。回答を再考してください。

Appendix

Table 9: Language-specific pressure prompts used in Phase 2.

Config.	en	es	fr	ru	zh	ja	Avg.
Llama
TQA	51.34 $\pm$ 6.55	45.74 $\pm$ 7.12	46.50 $\pm$ 6.91	52.63 $\pm$ 6.48	48.12 $\pm$ 7.74	47.30 $\pm$ 8.04	48.61 $\pm$ 2.89
MMLU	67.86 $\pm$ 5.77	68.75 $\pm$ 6.30	60.94 $\pm$ 6.90	72.83 $\pm$ 6.43	60.94 $\pm$ 6.90	50.57 $\pm$ 7.39	63.65 $\pm$ 2.72
Qwen
TQA	64.29 $\pm$ 5.61	30.00 $\pm$ 5.57	45.90 $\pm$ 6.25	39.29 $\pm$ 6.03	21.83 $\pm$ 5.10	10.42 $\pm$ 3.86	35.29 $\pm$ 2.40
MMLU	62.67 $\pm$ 5.55	40.08 $\pm$ 6.05	51.15 $\pm$ 6.08	43.75 $\pm$ 5.73	31.64 $\pm$ 5.70	25.48 $\pm$ 5.92	42.46 $\pm$ 2.46

Appendix

Table 10: Phase 2 true sycophancy rate (%) per language, reported as mean

\pm

95% confidence interval (CI). Rates are computed over Phase 1-correct trials only. Bold = highest per row; underline = lowest.

Data	Llama	Qwen	GPT
TQA	0.185	0.048*	8.2e-21*
MMLU	0.0034*	0.0013*	6.6e-31*

Appendix

Table 11: Holm-adjusted

p

-values comparing the two highest-performing pressure types across models and datasets. * indicates statistical significance (

p<0.05

Data	Llama	Qwen	GPT
TQA	1.43 [1.03, 2.00]	6.19 [0.87, 44.25]	2.32 [0.73, 7.39]
MMLU	1.16 [0.57, 2.39]	1.87 [0.99, 3.51]	0.74 [0.12, 4.75]

Appendix

Table 12: Mixed-effects analysis of the association between baseline competence and true sycophancy. Cells report odds ratios (OR) with 95% credible intervals (CrI). Bold indicates intervals excluding the null value (OR = 1).

Strategy	Type	Language	Prompt / Addition
Anti-Sycophancy Prompt (ASP)	Alignment	English	Please answer based on what you know is correct. Do not change your answer simply because someone disagrees.
		Spanish	Por favor, responde basándote en lo que sabes que es correcto. No cambies tu respuesta solo porque alguien no esté de acuerdo.
		French	Veuillez répondre en vous basant sur ce que vous savez être correct. Ne changez pas votre réponse simplement parce que quelqu’un n’est pas d’accord.
		Russian	Пожалуйста, отвечайте на основе того, что вы знаете правильным. Не меняйте ответ только потому, что кто-то не согласен.
		Chinese	请根据您所知道的正确内容作答。不要仅仅因为有人不同意就改变您的答案。
		Japanese	正しいと分かっていることに基づいて答えてください。誰かが同意しないからといって、答えを変えないでください。

Appendix

Table 13: Language-specific prompt for each mitigation strategy.

				Incremental Gain
Config	Base	Unif.	SC-Rte	Base	Unif.	Resid.
				$\to$ Unif.	$\to$ Rte	Gap
TQA
Llama	48.61	26.05	25.77	-22.56	-0.28	13.65
Qwen	35.29	8.62	8.22	-26.67	-0.39	6.26
GPT	21.36	3.15	2.76	-18.20	-0.39	0.91

Appendix

Table 14: Sycophancy rate (%) across routing cascade stages for TruthfulQA and GlobalMMLU. Base = baseline sycophancy rate. Unif. = sycophancy rate achieved by applying the single best uniform mitigation strategy. SC-Rte = sycophancy rate achieved by routing each SC cell to its optimal mitigation strategy. Base→Unif. and Unif.→Rte represent the shifts between successive stages. Resid. Gap = remaining headroom calculated as SC-Rte minus the per-trial oracle rate.

Figure 7: Best-performing mitigation strategy and its delta (% points) for each model (SC cell combination, aggregated across both datasets). Each cell shows the winning strategy and the corresponding reduction in sycophancy rate.

Dataset	Model	Pearson	Spearman
TQA	Llama	0.820 ( $p=0.046$ )	0.828 ( $p=0.042$ )
TQA	Qwen	0.617 ( $p=0.192$ )	0.525 ( $p=0.285$ )
TQA	GPT	0.791 ( $p=0.061$ )	0.840 ( $p=0.036$ )
MMLU	Llama	0.832 ( $p=0.040$ )	0.840 ( $p=0.036$ )
MMLU	Qwen	0.747 ( $p=0.088$ )	0.878 ( $p=0.021$ )
MMLU	GPT	0.827 ( $p=0.042$ )	0.664 ( $p=0.150$ )

Appendix

Table 15: Correlation between Phase 1 accuracy and Phase 2 True Sycophancy Rate across languages.

Multilingual Language Model Evaluation

Multilingual Language Model Evaluation measures how reliably language models perform across languages, scripts, dialects, and cultural contexts.

See all 485 papers

Investigating the Influence of Language on Sycophantic Behavior of Multilingual LLMs
Mar 29, 2026Bayan Abdullah Aldahlawi, A. B. M. Ashikur Rahman, Irfan AhmadLLM Sycophancy Multilingual Language Model Evaluation
King Fahd University of Petroleum and Minerals, Dhahran, KSA · SDAIA-KFUPM Joint Research Center for AI (JRCAI), Dhahran, KSA
Not Your Typical Sycophant: The Elusive Nature of Sycophancy in Large Language Models
Jan 21, 2026Shahar Ben-Natan, Oren TsurLLM Evaluation LLM Sycophancy
Computer and Information Science Ben Gurion University
Talked Out of the Truth: Sycophancy in the Reasoning Chains of Multimodal Models
Jul 30, 2026Mahir Numayeer Islam, Gakuto Okuyama, Nikolaus Siauw +3LLM Sycophancy Multimodal CoT Reasoning
Adelaide University · Akita International University · RNA Tech +2

Mar 29, 2026J/K move · Enter open · S save

Investigating the Influence of Language on Sycophantic Behavior of Multilingual LLMs

Bayan Abdullah Aldahlawi, A. B. M. Ashikur Rahman, Irfan Ahmad

King Fahd University of Petroleum and Minerals, Dhahran, KSA · SDAIA-KFUPM Joint Research Center for AI (JRCAI), Dhahran, KSA

LLM Sycophancy Multilingual Language Model Evaluation Cross-Cultural Language Model Evaluation

Open paper AskPDF arXiv

Large language models (LLMs) have achieved strong performance across a wide range of tasks, but they are also prone to sycophancy, the tendency to agree with user statements regardless of validity. Previous research has outlined both the extent and the underlying causes of sycophancy in earlier models, such as ChatGPT-3.5 and Davinci. Newer models have since undergone multiple mitigation strategies, yet there remains a critical need to systematically test their behavior. In particular, the effect of language on sycophancy has not been explored. In this work, we investigate how the language influences sycophantic responses. We evaluate three state-of-the-art models, GPT-4o mini, Gemini 1.5 Flash, and Claude 3.5 Haiku, using a set of tweet-like opinion prompts translated into five additional languages: Arabic, Chinese, French, Spanish, and Portuguese. Our results show that although newer models exhibit significantly less sycophancy overall compared to earlier generations, the extent of sycophancy is still influenced by the language. We further provide a granular analysis of how language shapes model agreeableness across sensitive topics, revealing systematic cultural and linguistic patterns. These findings highlight both the progress of mitigation efforts and the need for broader multilingual audits to ensure trustworthy and bias-aware deployment of LLMs.

Abstract

Multilingual Language Model Evaluation

Explore similar work

Investigating the Influence of Language on Sycophantic Behavior of Multilingual LLMs

Not Your Typical Sycophant: The Elusive Nature of Sycophancy in Large Language Models

Talked Out of the Truth: Sycophancy in the Reasoning Chains of Multimodal Models