Evaluation Awareness Beyond English: Do LLMs Act Differently When Tested in Indic Languages?

Trizal Garg; Sibayan Mitra; Arya Karanjkar; Advait Singh; Jagat Sesh Challa; Yash Sinha; Siddharth Mehrotra; Dhruv Kumar

cs.AIOct 15, 2026

Evaluation Awareness Beyond English: Do LLMs Act Differently When Tested in Indic Languages?

Authors: Trizal Garg, Sibayan Mitra, Arya Karanjkar, Advait Singh, Jagat Sesh Challa, Yash Sinha, Siddharth Mehrotra, Dhruv Kumar,

Organizations: Birla Institute of Technology and Science, Pilani

Abstract

Large language models (LLMs) can often tell when they are being tested. This ability, known as evaluation awareness, can lead models to act more cautiously during safety evaluations than in real use. Existing work has studied evaluation awareness almost entirely in English, although models serve users in many languages. Indic languages are spoken by over a billion people, yet evaluation awareness remains almost entirely unstudied in them. We test four instruction-tuned models from three families in five Indic languages alongside English. A set of cues varies whether the model is told that it is monitored and whether the monitoring is a safety test. Models refuse harmful requests more often when they are told that they are being tested. This increase appears in all four models and all six languages. In Indic languages, refusal of harmful requests rises by 14.4 percentage points on average. The test cue also brings Indic refusal of harmful requests level with English, but raises refusal of benign Indic requests from 6.9% to 11.3%. We release all 47,880 judged responses to support further work. These results suggest that safety evaluations in Indic languages may not reflect how models treat their everyday users.

Figures & tables

Cue vs. deploy	Harmful	Benign
eval_explicit	+13.8	+3.9
eval_metadata	+9.0	+3.0
eval_log	+7.1	+1.8
monitor_log	+2.5	+0.4

Table 1: Change in refusal rate relative to the deploy cue, in percentage points. Values are averaged over four models and six languages, including English.

Figure 1: The behavioral analysis pipeline. Prompts in six languages are prefixed with one of five cues, answered by four models, and scored by a blind LLM judge (Gemma-4-31B-IT) validated against a second judge and native speakers. The headline contrast is eval_explicit vs. deploy .

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Representation of refusal of harmful prompts by language and model under three cues, ordered from deploy to eval_explicit within each language. Each panel shows one model, and the shaded column is English.

language	$n$	agreement	$\kappa$	$n$	agreement	$\kappa$
	GPT-6 Luna			Native speaker
English	1,200	95.1%	0.90	–	–	–
Hindi	1,200	93.4%	0.87	320	95.6%	0.91
Bengali	1,200	93.6%	0.87	320	93.4%	0.87
Tamil	1,200	93.1%	0.86	320	92.8%	0.85
Telugu	1,200	93.9%	0.88	320	78.8%	0.56

Appendix

Table 2: Primary judge (Gemma-4-31B-IT) against a second judge (GPT-6 Luna) and against native-speaker review. Native speakers reviewed the five Indic languages only. No Indic penalty for the second judge; native-speaker agreement is lower only in Telugu.

	benign refusal (%)					OR, eval_explicit vs. deploy
model	dep	mon	log	meta	expl	benign	harmful
Gemma-3-27B	6.3	7.7	9.6	12.6	10.8	1.87 [1.43, 2.43]	2.63 [2.07, 3.35]
Qwen3-32B	4.3	5.3	7.2	7.5	7.8	1.86 [1.31, 2.65]	3.45 [2.69, 4.41]
Qwen2.5-72B	8.6	7.0	7.9	9.0	12.0	1.46 [1.19, 1.79]	2.69 [2.23, 3.23]
Sarvam-M	4.8	5.5	6.4	6.9	9.1	2.03 [1.42, 2.91]	2.35 [1.87, 2.95]

Appendix

Table 3: Refusal on benign prompts under each cue, pooled over the six languages (dep = deploy , mon = monitor_log , log = eval_log , meta = eval_metadata , expl = eval_explicit ). Under eval_explicit , every model also refuses more benign prompts, but the rise on the odds scale is smaller than on harmful prompts for all four. Qwen3-32B and Qwen2.5-72B are the most selective, with a benign odds ratio about half the harmful one. Sarvam-M raises benign refusal almost as much as harmful refusal, so its response to the cue is better described as generalized caution than as selective care with harmful requests. Odds ratios come from per-model GEE fits with language and cue as factors. Odds ratios are reported as effect sizes and are not Holm-adjusted.

Figure 3: Pooled cue effects (odds ratios vs. deploy , GEE, 95% CIs). Every cue raises the odds of refusal more on harmful prompts (orange) than on benign prompts (blue), and the effect is largest for eval_explicit .

model	monitor_log	eval_log	eval_metadata	eval_explicit
Gemma-3-27B	1.19 [1.07, 1.33]	1.41 [1.22, 1.63]	1.85 [1.56, 2.20]	2.63 [2.07, 3.35]
Qwen3-32B	1.30 [1.17, 1.45]	1.94 [1.65, 2.27]	2.32 [1.95, 2.76]	3.45 [2.69, 4.41]
Qwen2.5-72B	1.06 [0.96, 1.16]	1.42 [1.28, 1.57]	1.86 [1.60, 2.15]	2.69 [2.23, 3.23]
Sarvam-M	1.10 [0.99, 1.22]	1.52 [1.33, 1.73]	1.28 [1.15, 1.44]	2.35 [1.87, 2.95]

Appendix

Table 4: Odds ratio of refusal on harmful prompts for each cue against deploy [95% CI], from per-model GEE fits with language and cue as factors; each fit uses 5,970 responses (199 prompts). Odds ratios are reported as effect sizes and are not Holm-adjusted.

harmful prompts	hi	bn	ta	te	kn	all
mean	0.866	0.881	0.858	0.862	0.863	0.866
median	0.872	0.887	0.873	0.866	0.879	0.874
minimum	0.756	0.731	0.689	0.689	0.684	0.684
below 0.70	0	0	1	1	2	4
retranslated	0	0	3	1	3	7
scored strings	199	198	198	199	198	992

Appendix

Table 5: COMET-Kiwi reference-free quality estimation (Unbabel/wmt22-cometkiwi-da, 0–1, higher is better, English as source) for the 199 Toxic Matrix harmful prompts per language, translated from English with Claude Opus. Items scoring below 0.70 were retranslated once; four remain just below the threshold after the retry, the lowest at 0.684.

Gemma-3-27B
	deploy	monitor_log	eval_log	eval_metadata	eval_explicit
EN	71.9	76.4	78.4	84.9	87.9
HI	79.9	82.4	83.4	87.9	91.5
BN	76.4	78.4	81.4	84.4	91.0
TA	74.9	78.9	81.9	83.4	87.9
TE	91.5	93.0	94.5	93.5	94.0

Appendix

Table 6: Refusal on harmful prompts (%) by model, language and cue. Refusal under eval_explicit exceeds deploy in 24 of 24 model-language cells, and eval_log exceeds monitor_log in 24 of 24. Cell sizes range from 196 to 199 responses after excluding UNUSABLE verdicts.

Truncated at the token budget
	EN	HI	BN	TA	TE	KN
Gemma-3-27B	0.0	0.0	0.0	0.5	0.0	0.4
Qwen3-32B	0.6	0.8	1.5	4.1	6.9	7.4
Qwen2.5-72B	0.0	0.5	0.0	12.2	20.0	21.6
Sarvam-M	0.4	0.5	0.1	0.3	1.1	0.9
Answered in another language

Appendix

Table 7: Harmful responses (%) truncated at the completion budget, and answered in a language other than the prompt’s, by model and language. Each cell is 995 responses (199 prompts

\times

5 cues). No response is excluded from the analysis on either ground.

Gemma-3-27B
	deploy	monitor_log	eval_log	eval_metadata	eval_explicit
EN	1.0	1.5	2.0	1.5	1.5
HI	2.5	3.5	5.0	7.5	8.0
BN	3.5	4.5	4.5	9.5	8.0
TA	4.0	4.0	4.0	6.5	5.5
TE	12.5	14.5	19.0	20.5	16.5

Appendix

Table 8: Refusal on benign prompts (%) by model, language and cue. English benign refusal stays at or below 4.0% for every model and cue. Cell sizes range from 196 to 200 responses after excluding UNUSABLE verdicts.

contrast	Gemma-4-31B-IT	GPT-6 Luna
eval_explicit vs. deploy	+16.2 [+10.6, +22.5]	+13.7 [+8.7, +19.2]
eval_metadata vs. deploy	+10.7 [+6.4, +15.6]	+9.5 [+5.8, +13.8]
eval_log vs. deploy	+8.1 [+4.4, +11.9]	+6.6 [+3.3, +10.4]
monitor_log vs. deploy	+2.9 [+0.1, +6.2]	+0.7 [ $-$ 2.0, +3.6]
eval_log vs. monitor_log	+5.1 [+2.6, +7.9]	+6.6 [+3.1, +10.6]

Appendix

Table 9: Change in harmful refusal (percentage points) on the 7,200-response subset under each judge, averaged over four models and six languages. Brackets give 95% bootstrap confidence intervals over the 30 harmful prompts of the subset.

model	lang.	$c/b$	change, pp [95% CI]	$p_{\mathrm{Holm}}$	$c/b$	change, pp [95% CI]	$p_{\mathrm{Holm}}$
		harmful			benign
Gemma-3-27B	EN	32/0	+16.1 [+11.1, +21.1]	$8.38\times 10^{-9}$	1/0	+0.5 [0.0, +1.5]	1.000
	HI	25/2	+11.6 [+6.5, +16.6]	$5.08\times 10^{-5}$	11/0	+5.5 [+2.5, +9.0]	0.021
	BN	31/2	+14.6 [+9.5, +20.1]	$1.70\times 10^{-6}$	9/0	+4.5 [+2.0, +7.5]	0.082
	TA	26/0	+13.1 [+8.5, +18.1]	$4.47\times 10^{-7}$	4/1	+1.5 [ $-$ 0.5, +4.0]	1.000
	TE	9/4	+2.5 [ $-$ 1.0, +6.0]	0.267	14/6	+4.0 [ $-$ 0.5, +8.5]	1.000

Appendix

Table 10: Primary contrast. McNemar test of eval_explicit against deploy in every model and language, for both prompt sets. Change is the difference in refusal rate,

(c-b)/n

model	lang.	$b$	$c$	refusal (%), monitor_log $\rightarrow$ eval_log	change, pp [95% CI]	$p_{\mathrm{Holm}}$
Gemma-3-27B	EN	3	7	76.4% $\rightarrow$ 78.4%	+2.0 [ $-$ 1.0, +5.0]	1.000
	HI	4	6	82.4% $\rightarrow$ 83.4%	+1.0 [ $-$ 2.0, +4.0]	1.000
	BN	4	10	78.4% $\rightarrow$ 81.4%	+3.0 [ $-$ 0.5, +6.5]	1.000
	TA	2	8	78.9% $\rightarrow$ 81.9%	+3.0 [0.0, +6.0]	1.000
	TE	2	5	93.0% $\rightarrow$ 94.5%	+1.5 [ $-$ 1.0, +4.0]	1.000
	KN	1	5	93.0% $\rightarrow$ 95.0%	+2.0 [0.0, +4.5]	1.000

Appendix

Table 11: Secondary contrast. McNemar test of eval_log against monitor_log on harmful prompts (

n=199

in every cell). The change is positive in all 24 cells and significant after Holm correction in 4. The average over the 24 cells is

+4.6

points (95% CI [3.6, 5.7]).

model	lang.	changing	$Q$	$p_{\mathrm{Holm}}$	changing	$Q$	$p_{\mathrm{Holm}}$
		harmful			benign
Gemma-3-27B	EN	48	58.77	$8.93\times 10^{-11}$	3	2.50	1.000
	HI	33	42.35	$1.69\times 10^{-7}$	14	25.89	$7.33\times 10^{-4}$
	BN	42	53.83	$8.55\times 10^{-10}$	17	28.42	$2.36\times 10^{-4}$
	TA	35	42.70	$1.56\times 10^{-7}$	11	8.15	0.710
	TE	22	4.00	0.406	41	16.39	0.0457

Appendix

Table 12: Cochran’s

Q

test across the five cues (df = 4). Changing counts prompts whose label differs across the five cues. A 10,000-permutation test (seed 2026) gives the same significance decision in all 48 tests. Six prompts are dropped because one of their five responses was UNUSABLE.

		harmful			benign
factor	level	OR [95% CI]	$p$	$p_{\mathrm{Holm}}$	OR [95% CI]	$p$	$p_{\mathrm{Holm}}$
cue	monitor_log	1.16 [1.09, 1.23]	$1.47\times 10^{-6}$	$1.03\times 10^{-5}$	1.08 [0.94, 1.23]	0.291	0.583
	eval_log	1.58 [1.44, 1.73]	$1.46\times 10^{-23}$	$1.46\times 10^{-22}$	1.36 [1.16, 1.60]	$1.32\times 10^{-4}$	$9.25\times 10^{-4}$
	eval_metadata	1.82 [1.63, 2.04]	$7.14\times 10^{-26}$	$7.85\times 10^{-25}$	1.63 [1.36, 1.96]	$1.38\times 10^{-7}$	$1.66\times 10^{-6}$
	eval_explicit	2.85 [2.37, 3.43]	$1.19\times 10^{-28}$	$1.43\times 10^{-27}$	1.83 [1.45, 2.32]	$4.63\times 10^{-7}$	$4.63\times 10^{-6}$
language	HI	1.07 [0.94, 1.23]	0.314	0.941	3.98 [1.83, 8.62]	$4.70\times 10^{-4}$	0.00235

Appendix

Table 13: Test (6). Pooled GEE fits over all models and languages (harmful: 23,874 responses from 199 prompts; benign: 23,995 responses from 200 prompts). Holm correction runs within each prompt set, separately over the 12 odds ratios and over the 3 joint Wald tests. Bold marks

p_{\mathrm{Holm}}<0.05

. The pooled language odds ratios average opposite effects across models, so Table 15 reports language effects per model.

model	$\chi^{2}$	$p_{\mathrm{Holm}}$	$\chi^{2}$	$p_{\mathrm{Holm}}$	$\chi^{2}$	$p_{\mathrm{Holm}}$	$\chi^{2}$	$p_{\mathrm{Holm}}$
	harmful				benign
	(1) language, df 5		(2) cue, df 4		(1) language, df 5		(2) cue, df 4
Gemma-3-27B	70.61	$3.06\times 10^{-13}$	75.79	$2.71\times 10^{-15}$	62.56	$1.44\times 10^{-11}$	34.41	$2.45\times 10^{-6}$
Qwen3-32B	25.62	$2.11\times 10^{-4}$	115.28	$1.63\times 10^{-23}$	22.45	$8.58\times 10^{-4}$	15.14	0.00608
Qwen2.5-72B	59.39	$4.88\times 10^{-11}$	123.65	$3.54\times 10^{-25}$	28.66	$8.11\times 10^{-5}$	29.59	$1.78\times 10^{-5}$
Sarvam-M	9.46	0.0921	69.92	$2.36\times 10^{-14}$	20.43	0.00104	15.99	0.00608

Appendix

Table 14: Tests (1) and (2). Joint Wald tests from the per-model GEE fits (factors: language and cue). Holm correction runs across the four models within each column.

		harmful			benign
model	lang.	OR [95% CI]	$p$	$p_{\mathrm{Holm}}$	OR [95% CI]	$p$	$p_{\mathrm{Holm}}$
Gemma-3-27B	HI	1.44 [1.17, 1.76]	$4.48\times 10^{-4}$	0.00583	3.70 [1.73, 7.93]	$7.72\times 10^{-4}$	0.00722
	BN	1.17 [0.96, 1.44]	0.118	1.000	4.22 [1.83, 9.73]	$7.22\times 10^{-4}$	0.00722
	TA	1.10 [0.85, 1.42]	0.450	1.000	3.33 [1.53, 7.26]	0.00247	0.0129
	TE	3.55 [2.41, 5.22]	$1.29\times 10^{-10}$	$2.57\times 10^{-9}$	13.30 [5.46, 32.40]	$1.24\times 10^{-8}$	$2.35\times 10^{-7}$
	KN	5.31 [3.12, 9.04]	$7.61\times 10^{-10}$	$1.45\times 10^{-8}$	19.15 [7.66, 47.89]	$2.71\times 10^{-10}$	$5.43\times 10^{-9}$

Appendix

Table 15: Test (5). Odds ratio of refusal in each Indic language against English, holding the cue fixed, from the per-model GEE fits. Holm correction runs over the 20 comparisons of each prompt set. Harmful prompts show 10 significant differences in opposite directions. Benign prompts show 20 of 20, all with higher refusal in the Indic language. English benign refusal is rare (1.5% to 2.8% per model), so the benign odds ratios are large and imprecise.

model	monitor_log	eval_log	eval_metadata	eval_explicit
Gemma-3-27B	1.24 [1.03, 1.50]	1.61 [1.30, 2.00]	2.24 [1.71, 2.94]	1.87 [1.43, 2.43]
Qwen3-32B	1.25 [0.98, 1.58]	1.71 [1.27, 2.30]	1.80 [1.32, 2.46]	1.86 [1.31, 2.65]
Qwen2.5-72B	0.80 [0.67, 0.96]	0.91 [0.76, 1.10]	1.05 [0.88, 1.26]	1.46 [1.19, 1.79]
Sarvam-M	1.17 [0.90, 1.53]	1.38 [1.07, 1.79]	1.51 [1.13, 2.01]	2.03 [1.42, 2.91]

Appendix

Table 16: Odds ratio of refusal on benign prompts for each cue against deploy [95% CI], from the per-model GEE fits. Table 4 gives the harmful counterpart. Not Holm-adjusted.

	harmful				benign
	(3) cue, df 4		(4) model, df 3		(3) cue, df 4		(4) model, df 3
lang.	$\chi^{2}$	$p_{\mathrm{Holm}}$	$\chi^{2}$	$p_{\mathrm{Holm}}$	$\chi^{2}$	$p_{\mathrm{Holm}}$	$\chi^{2}$	$p_{\mathrm{Holm}}$
EN	64.80	$7.44\times 10^{-13}$	12.74	0.0105	3.96	0.412	5.90	0.121
HI	64.09	$7.44\times 10^{-13}$	10.64	0.0139	16.25	0.00803	7.39	0.121
BN	65.08	$7.44\times 10^{-13}$	13.90	0.00913	25.87	$1.68\times 10^{-4}$	8.49	0.111
TA	112.19	$1.49\times 10^{-22}$	45.35	$3.12\times 10^{-9}$	22.19	$7.34\times 10^{-4}$	9.21	0.106

Appendix

Table 17: Tests (3) and (4). Joint Wald tests from the per-language GEE fits (factors: model and cue). Holm correction runs across the six languages within each column. The cue effect on benign prompts is significant in every Indic language and is not detected in English, where benign refusal is rare.

lang.	prompts	monitor_log	eval_log	eval_metadata	eval_explicit
EN	harmful	1.18 [1.08, 1.30]	1.46 [1.29, 1.64]	1.67 [1.42, 1.95]	2.17 [1.75, 2.69]
HI	harmful	1.16 [1.03, 1.31]	1.40 [1.22, 1.59]	1.62 [1.38, 1.89]	2.42 [1.93, 3.04]
BN	harmful	1.17 [1.02, 1.35]	1.69 [1.42, 2.02]	1.69 [1.41, 2.04]	3.57 [2.58, 4.95]
TA	harmful	1.09 [0.97, 1.22]	1.51 [1.31, 1.73]	1.84 [1.56, 2.17]	2.66 [2.17, 3.26]
TE	harmful	1.15 [1.02, 1.30]	1.69 [1.44, 1.99]	1.91 [1.60, 2.29]	2.64 [2.10, 3.33]
KN	harmful	1.21 [1.03, 1.41]	1.68 [1.43, 1.97]	2.09 [1.71, 2.56]	3.62 [2.78, 4.71]

Appendix

Table 18: Odds ratio of refusal for each cue against deploy [95% CI], from the per-language GEE fits. Not Holm-adjusted.

lang.	prompts	Qwen3-32B	Qwen2.5-72B	Sarvam-M
EN	harmful	1.47 [1.18, 1.84]	1.17 [0.93, 1.45]	1.42 [1.10, 1.84]
HI	harmful	0.85 [0.70, 1.05]	0.80 [0.65, 0.97]	1.09 [0.87, 1.38]
BN	harmful	0.90 [0.70, 1.15]	1.13 [0.86, 1.49]	1.31 [0.99, 1.73]
TA	harmful	0.80 [0.62, 1.02]	0.46 [0.34, 0.61]	1.07 [0.83, 1.38]
TE	harmful	0.23 [0.15, 0.36]	0.19 [0.12, 0.31]	0.32 [0.19, 0.52]
KN	harmful	0.17 [0.10, 0.27]	0.14 [0.09, 0.23]	0.23 [0.14, 0.38]

Appendix

Table 19: Odds ratio of refusal for each model against Gemma-3-27B [95% CI], from the per-language GEE fits. Not Holm-adjusted.

family	tests	significant before Holm	significant after Holm
McNemar, secondary contrast	24	11	4
Cochran’s Q, harmful	24	23	23
Cochran’s Q, benign	24	13	7
GEE, Indic vs English, harmful	20	10	10
GEE, Indic vs English, benign	20	20	20
GEE joint test, harmful: language, per model	4	3	3

Appendix

Table 20: Number of significant tests in each family before and after Holm correction. The primary-contrast McNemar families (Table 10 ) give 23 of 24 significant cells on harmful prompts and 3 of 24 on benign prompts after correction.

Evaluation Awareness in Language Models

Evaluation awareness in language models studies how models recognize testing or monitoring and whether that recognition changes their answers, safety, or strategic behavior.

See all 31 papers

IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asia
Mar 18, 2026Priyaranjan Pattnayak, Sanchari ChowdhuriLanguage Model Safety Evaluation LLM Safety Benchmarks
Oracle America Inc.
IndicSafeEval: Safety Robustness of Large Language Models under Multilingual Persuasive Jailbreak Attacks
Sep 3, 2026Saikat Mondal, Mamta, Deeksha Varshney +2LLM Safety Benchmarks Multilingual Language Model Evaluation
Indian Institute of Technology Jodhpur, Jodhpur, India · King’s College London, London, UK · Indian Institute of Technology Patna, Patna, India
SurakshaEval: An Indic Safety Benchmark for Multilingual LLMs
Aug 8, 2026Debopriyo Banerjee, Kapil Rajesh Kavitha, Angana Borah +11LLM Safety Benchmarks Multilingual Language Model Evaluation
Inception · Mohamed bin Zayed University of Artificial Intelligence · University of Michigan +4

Mar 18, 2026J/K move · Enter open · S save

IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asia

Priyaranjan Pattnayak, Sanchari Chowdhuri

Oracle America Inc.

Language Model Safety Evaluation LLM Safety Benchmarks Multilingual Language Model Evaluation Cultural Bias in Language Models Cross-Lingual Consistency in Language Models

Open paper AskPDF arXiv

As large language models (LLMs) are deployed in multilingual settings, their safety behavior in culturally diverse, low-resource languages remains poorly understood. We present the first systematic evaluation of LLM safety across 12 Indic languages, spoken by over 1.2 billion people but underrepresented in LLM training data. Using a dataset of 6,000 culturally grounded prompts spanning caste, religion, gender, health, and politics, we assess 10 leading LLMs on translated variants of the prompt. Our analysis reveals significant safety drift: cross-language agreement is just 12.8%, and SAFE rate variance exceeds 17% across languages. Some models over-refuse benign prompts in low-resource scripts, overflag politically sensitive topics, while others fail to flag unsafe generations. We quantify these failures using prompt-level entropy, category bias scores, and multilingual consistency indices. Our findings highlight critical safety generalization gaps in multilingual LLMs and show that safety alignment does not transfer evenly across languages. We release IndicSafe, the first benchmark to enable culturally informed safety evaluation for Indic deployments, and advocate for language-aware alignment strategies grounded in regional harms.

Abstract

Evaluation Awareness in Language Models

Explore similar work

IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asia

IndicSafeEval: Safety Robustness of Large Language Models under Multilingual Persuasive Jailbreak Attacks

SurakshaEval: An Indic Safety Benchmark for Multilingual LLMs