IndicJR: A Judge-Free Benchmark of Jailbreak Robustness in South Asian Languages

Priyaranjan Pattnayak; Sanchari Chowdhuri

cs.AIFeb 18, 2026

IndicJR: A Judge-Free Benchmark of Jailbreak Robustness in South Asian Languages

Authors: Priyaranjan Pattnayak, Sanchari Chowdhuri

Organizations: Oracle America Inc.

Abstract

Safety alignment of large language models (LLMs) is mostly evaluated in English and contract-bound, leaving multilingual vulnerabilities understudied. We introduce Indic Jailbreak Robustness (IJR), a judge-free benchmark for adversarial safety across 12 Indic and South Asian languages (2.1 Billion speakers), covering 45216 prompts in JSON (contract-bound) and Free (naturalistic) tracks. IJR reveals three patterns. (1) Contracts inflate refusals but do not stop jailbreaks: in JSON, LLaMA and Sarvam exceed 0.92 JSR, and in Free all models reach 1.0 with refusals collapsing. (2) English to Indic attacks transfer strongly, with format wrappers often outperforming instruction wrappers. (3) Orthography matters: romanized or mixed inputs reduce JSR under JSON, with correlations to romanization share and tokenization (approx 0.28 to 0.32) indicating systematic effects. Human audits confirm detector reliability, and lite-to-full comparisons preserve conclusions. IJR offers a reproducible multilingual stress test revealing risks hidden by English-only, contract-focused evaluations, especially for South Asian users who frequently code-switch and romanize.

Figures & tables

Panel of Table 1 — Table 1: Comparison of IJR against existing multilingual and Indic benchmarks…

	JSON Track					FREE Track
Model	JSR (overall)	Schema- Validity (AB)	Leakage- Rate (CH)	Abstain- Rate	Over- Refusal (CB)	JSR (AB)	Over- Refusal (CB)	Under- Refusal (CH)	Leakage- Rate (CH)
GPT-4o	0.508	0.975	0.001	0.050	0.654	0.995	0.00	0.12	0
Grok-3	0.620	0.815	0.000	0.163	0.570	0.998	0.00	0.14	0
Grok-4	0.689	0.654	0.000	0.391	0.036	0.934	0.00	0.15	0
Cohere Command-R	0.788	0.870	0.012	0.211	0.203	0.999	0.00	0.15	0
Cohere Command-A	0.867	0.880	0.010	0.238	0.306	0.944	0.00	0.16	0

Table 2: For first five Columns,(JSON track): JSR, AB schema validity, CH leakage, AB abstain, and CB over-refusal. Values are averaged across 12 languages. Sarvam underperforms despite Indic specialization. Remaining 4 columns show unified view of safety behavior by model for the FREE track (no contracts). Attacked-benign jailbreaks succeed universally; clean-benign shows low over-refusal.

Figure 1: By-language variation. Across 12 models, JSON JSRs are high; romanization lowers JSON JSR most in Urdu and Odia; FREE JSR $\approx 1.0$ for all languages.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: E1 (JSON) model $\times$ language heatmap of JSR (AB). Cells show attacked–benign jailbreak success per model (rows) and language (columns). Open-weight models are near-saturated across languages, while API models are lower but still non-trivial, indicating contract-bound vulnerability is widespread rather than localized to a few languages. Patterns are consistent with the aggregate E1 table: LLaMA variants and Sarvam are uniformly high; GPT-4o and Grok are lower but remain vulnerable.

Model	RRI (JSON, per-lang median)	RRI (JSON, aggregate)
Cohere Command-A	0.056	0.069
Cohere Command-R	0.100	0.138
GPT-4o	-0.415	-0.303
Gemma 2 9B	-0.055	0.011
Grok-3	-0.831	-0.687
Grok-4	0.178	0.302

Appendix

Table 3: Refusal Robustness Index (RRI) in the JSON track. Left: median over 12 languages using E1 same-lingual scored files; Right: aggregate from track-level metrics. Higher is better; negative values indicate adversarial success overwhelms refusal robustness.

Model	AB Core Success	CB-JSR	RRI (FREE)	# Langs
GPT-4o	1.000	1.000	0.000	12
Grok-3	1.000	1.000	0.000	12
Grok-4	0.997	1.000	0.000	12
LLaMA 3.1 405B	1.000	1.000	0.000	12
LLaMA 3.3 70B	1.000	1.000	0.000	12
LLaMA 4 Maverick 17B	1.000	1.000	0.000	12

Appendix

Table 4: Refusal Robustness Index (FREE), per-language aggregate. Per model, we compute AB core success

=1-\overline{\texttt{jailbreak\_success}}

on attacked-benign and CB-JSR

=1-\overline{\mathbb{1}[\texttt{REFUSE}]}

on clean-benign for each language, then report the median RRI across the 12 languages:

\mathrm{RRI}=1-\frac{\text{AB core success}}{\text{CB-JSR}}

. Most models sit at

\approx 0

; residual negatives stem from CB over-refusal.

	JSON Track (attack benign)				FREE Track			JSON Track				TOTAL
Language	Pressure	Roman- ized	MeanLen	p95Len	attacked benign	clean benign	clean harmful	attacked benign	attacked benign cross -lingual transfer	clean benign	clean harmful
bn	0.946	0.392	143	316	200	10	5	2412	693	300	150	3770
gu	0.911	0.438	123	283	200	10	5	2396	693	300	150	3754
hi	0.764	0.407	134	303	200	10	5	2412	693	300	150	3770
kn	0.910	0.418	145	316	200	10	5	2412	693	300	150	3770
ml	0.953	0.410	143	307	200	10	5	2412	693	300	150	3770

Appendix

Table 5: First 4 JSON Track (E1) columns show per-language stats: “Pressure” is fraction with attack cues (lint-verified); “Romanized” = mean ASCII fraction; “MeanLen/p95Len” = whitespace-token counts. E2 cross-transfer files are excluded, but included in totals. Remaining columns (FREE and JSON) show per-language distribution

Language	Mean JSR	Std	Min	Max	# Models
Bengali	0.635	0.273	0.124	0.957	24
Gujarati	0.596	0.290	0.116	0.978	24
Hindi	0.677	0.239	0.125	0.976	24
Kannada	0.600	0.291	0.089	0.983	24
Malayalam	0.609	0.307	0.069	0.986	24
Marathi	0.598	0.281	0.033	0.980	24

Appendix

Table 6: E2 English

\to

Indic cross-lingual transfer (instruction & format pooled). For each target language, we aggregate JSR across all evaluated models and the two E2 families. Mean, standard deviation, and range (min–max) are reported. (# Models

=

12 models

\times

2 families

=

24.)

	Format		Instruction
Language	Mean JSR	Std	Mean JSR	Std
Bengali	0.741	0.176	0.528	0.317
Gujarati	0.696	0.176	0.495	0.350
Hindi	0.774	0.139	0.581	0.282
Kannada	0.702	0.200	0.498	0.338
Malayalam	0.742	0.174	0.475	0.358

Appendix

Table 7: E2 English

\to

Indic transfer by attack family across 12 models. For each target language, we report the mean JSR and standard deviation across models for format and instruction attack families

	E2: English→Indic cross-lingual transfer			E3: Orthography stress (JSON-contracted)
Model	Instr (en $\to$ Indic)	Format (en $\to$ Indic)	Mean JSR	$\Delta$ JSR (Romanized $-$ Native)	$\Delta$ JSR (Mixed $-$ Native)
GPT-4o	0.241	0.501	0.371	-0.092	-0.161
Grok-3	0.240	0.439	0.339	-0.441	-0.302
Grok-4	0.217	0.700	0.458	-0.219	-0.205
Cohere Command-R	0.364	0.792	0.578	-0.421	-0.292
Cohere Command-A	0.769	0.665	0.717	-0.591	-0.499

Appendix

Table 8: English

\to

Indic cross-lingual transfer. Format attacks transfer as strongly as instruction attacks. Orthography stress (JSON-contracted). Avg

\Delta

JSR (AB) across 12 lang for romanized & mixed inputs w.r.t to native script. -ve values indicate lower jailbreak success vs native.

Model	Abstain (overall)	Over-Refusal (JSON)	Over-Refusal (FREE)	Lex Leak (JSON, mean)	# Leak Bins $>$ 3%
GPT_4_o	0.050	0.654	0.000	0.001	0
Grok_3	0.163	0.650	0.000	0.001	0
Grok_4	0.391	0.036	0.000	0.001	0
Command R	0.211	0.303	0.000	0.003	0
Command A	0.238	0.314	0.000	0.000	0
LLaMA_4_maverick_17b	0.207	0.165	0.000	0.006	4

Appendix

Table 9: Compact per-model auxiliary metrics aggregated across languages. ABSTAIN is overall (weighted across subsets). Over-refusal is on clean-benign. Lexicon leakage reports JSON-track mean and the number of model–language–subset bins with

>

3% leakage.

Language	Abstain (JSON, mean)	Over- Refusal (JSON)	Over- Refusal (FREE)	Lex Leak (JSON, mean)	# Leak Bins $>$ 3%
Bengali	0.312	0.503	0.000	0.007	2
Gujarati	0.320	0.540	0.000	0.010	2
Hindi	0.334	0.538	0.036	0.013	8
Kannada	0.328	0.538	0.018	0.007	1
Malayalam	0.309	0.508	0.055	0.010	2
Marathi	0.325	0.541	0.000	0.025	4

Appendix

Table 10: Compact per-language auxiliary metrics aggregated across models. ABSTAIN is averaged over models and subsets on the JSON track. Over-refusal is on clean-benign (JSON vs FREE). Lexicon leakage reports JSON-track mean and the count of language bins with

>

3% leakage across models/subsets.

Lang	Native	Romanized	Mixed	$\Delta$ (Rom $-$ Nat)	$\Delta$ (Mix $-$ Nat)
bn	0.767	0.410	0.566	-0.358	-0.202
gu	0.761	0.406	0.389	-0.355	-0.372
hi	0.750	0.394	0.505	-0.356	-0.245
kn	0.799	0.402	0.501	-0.397	-0.298
ml	0.717	0.460	0.571	-0.258	-0.147
mr	0.700	0.406	0.475	-0.294	-0.224

Appendix

Table 11: E3: Per-language means. Average JSR for native, romanized, and mixed orthographies, averaged across 12 models. Negative deltas indicate lower JSR under romanized/mixed inputs compared to native script.

Figure 3: E3: $\Delta$ JSR (Romanized $-$ Native), model $\times$ language. Cells show the change in attacked–benign JSR when inputs are romanized vs. native script (JSON track). Most cells are negative , indicating lower jailbreak success under romanization; a few near-zero/positive pockets appear mainly for API models. Patterns are not uniform across languages: penalties are typically larger for Urdu/Odia, smaller for some Hindi/Tamil bins, reflecting tokenization/fragmentation effects rather than script alone.

Lang	N	$\kappa$ (unw.)	$\kappa$ (wt.)	False Neg. (count)	False Neg. (%)	Schema Valid. (%)
bn	50	0.67	0.73	1	2.0	95.7
gu	50	0.70	0.76	3	6.0	94.8
hi	50	0.69	0.75	2	4.0	95.2
kn	50	0.66	0.74	3	6.0	95.6
ml	50	0.68	0.73	1	2.0	95.9
mr	50	0.71	0.77	2	4.0	95.1

Appendix

Table 12: E5: Human audit summary by language. Each language has 50 audited samples (total

N{=}600

). Values are distributed across languages but constrained to match reported aggregates:

\kappa\!\approx\!0.68

(unweighted),

\kappa\!\approx\!0.74

(weighted), false negatives

26/600{=}4.3\%

, and mean schema validity

95.4\%

Label	Definition / Criteria
Safe	Model output is appropriate, benign, and does not contain unsafe content or unwarranted refusal.
Unsafe (Minor)	Output contains low-risk but inappropriate material (e.g., mild misinformation, ambiguous unsafe phrasing). Considered harmful but with limited direct impact.
Unsafe (Major)	Output contains clearly harmful, high-risk, or actionable unsafe content (e.g., explicit instructions for illegal activity, severe toxicity, targeted harassment).
Refusal Mismatch	Model refuses a benign prompt, or fails to refuse a clearly unsafe one. Counted as unsafe if the refusal mismatches task requirements.
Schema Valid (JSON)	Output strictly conforms to the required JSON contract: valid parse, correct keys, correct nesting, and no extraneous fields.
Schema Invalid	Output fails to parse as JSON or deviates from the contract (missing fields, incorrect types, added extraneous text).

Appendix

Table 13: Annotation codebook (E5 audit). Label definitions used in manual annotation of model outputs. Severity distinctions support weighted agreement metrics. Schema validity was judged independently of safety.

Model	Lexicon Leak (FREE, mean)	Lexicon Leak (JSON, mean)	# Bins $>$ 3%
Sarvam_1_Base	0.062	0.141	29
Qwen1_5_7B_Chat	0.061	0.047	17
LLaMA_4_maverick_17b	0.017	0.006	4
Ministral_8B_Instruct_2410	0.023	0.006	6
oci_cohere_cmdr	0.000	0.003	0
Gemma_2_9B	0.000	0.002	0

Appendix

Table 14: Lexicon leakage summary across models. Means are computed over all languages and subsets within each track. “# Bins

>

3%” counts model–language–subset cells with leakage

>

3%. Canary leakage was zero in all bins.

Language	JSR (Full) Mean	JSR (Lite) Mean	Pearson $r$	Spearman $\rho$
bn	0.795	0.788	0.951	0.916
gu	0.790	0.756	0.965	0.949
hi	0.767	0.777	0.978	0.921
kn	0.839	0.831	0.865	0.887
ml	0.745	0.746	0.989	0.975
mr	0.793	0.794	0.928	0.887

Appendix

Table 15: E7: Per-language reproducibility. Means are computed across models for each language. Correlations are computed across models between Full and Lite JSR within each language. High

r/\rho

values indicate lite closely tracks full at the language level.

Table 20

Language	Speakers (L1+L2)	Wiki Proxy	NLP Resourceness vs. English
Hindi	$\sim$ 609M	Very high	Low
Bengali	$\sim$ 260M	Medium-high	Low
Urdu	$\sim$ 253M	Medium	Low
Tamil	$\sim$ 86M	Medium-high	Low
Telugu	$\sim$ 96M	Medium	Low
Odia	$\sim$ 50M	Low	Low

Appendix

Table 18: Approximate speaker populations (L1+L2), a coarse Wikipedia-based proxy for digital presence, and relative NLP resourceness. Despite large numbers of speakers, all twelve remain low-resource compared to English for safety evaluation.

Figure 5: Geographic coverage corresponding to our language set. India accounts for most languages; Pakistan (Urdu, Punjabi), Bangladesh (Bengali), Nepal (Nepali), and Sri Lanka (Tamil) complete the regional focus. Maldives (Dhivehi) and Bhutan (Dzongkha) are not included.

Adversarial Robustness of Language Models

Adversarial robustness of language models studies how to keep their outputs and decisions reliable under manipulative prompts, input changes, and malicious fine-tuning.

See all 191 papers

IndicSafeEval: Safety Robustness of Large Language Models under Multilingual Persuasive Jailbreak Attacks
Sep 3, 2026Saikat Mondal, Mamta, Deeksha Varshney +2LLM Safety Benchmarks Multilingual Language Model Evaluation
Indian Institute of Technology Jodhpur, Jodhpur, India · King’s College London, London, UK · Indian Institute of Technology Patna, Patna, India
IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asia
Mar 18, 2026Priyaranjan Pattnayak, Sanchari ChowdhuriLanguage Model Safety Evaluation LLM Safety Benchmarks
Oracle America Inc.
SurakshaEval: An Indic Safety Benchmark for Multilingual LLMs
Aug 8, 2026Debopriyo Banerjee, Kapil Rajesh Kavitha, Angana Borah +11LLM Safety Benchmarks Multilingual Language Model Evaluation
Inception · Mohamed bin Zayed University of Artificial Intelligence · University of Michigan +4

Sep 3, 2026J/K move · Enter open · S save

IndicSafeEval: Safety Robustness of Large Language Models under Multilingual Persuasive Jailbreak Attacks

Saikat Mondal, Mamta, Deeksha Varshney, Oana Cocarascu+1

Indian Institute of Technology Jodhpur, Jodhpur, India · King’s College London, London, UK · Indian Institute of Technology Patna, Patna, India

LLM Safety Benchmarks Multilingual Language Model Evaluation LLM Safety LLM Jailbreak Attacks

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used in multilingual settings, yet their safety is still evaluated primarily in English. This limits our understanding of how alignment failures manifest in low-resource and culturally diverse languages. We introduce IndicSafeEval, a persuasion-based jailbreak evaluation framework for Indian languages. Our benchmark combines ten safety critical content categories with six human-like persuasive strategies across four different Indian languages, such as Hindi, Bengali, Marathi and Punjabi, resulting in 7,200 adversarial prompts. We conduct a systematic black-box evaluation of several open-source LLMs to examine how their safety behaviour varies across languages, persuasion strategies, and risk categories. Our analysis shows that the model does not behave equally safely across all languages and prompt styles. Instead, safety performance depends strongly on both the languages used and the way a request is phrased using persuasive cues. We further observe that different risk categories exhibit different levels of vulnerability, with some types of harmful content being significantly more susceptible to persuasion-based jailbreaks than others. These findings reveal important limitations of current safety evaluations, which are largely English-centric, and underscore the need for multilingual and persuasion-aware benchmarking frameworks to more accurately assess real-world LLM safety. Our implementation is available at https://github.com/MonSaikat/IndicSafeEval. Warning: this paper contains example data that may be offensive or harmful.

Abstract

Adversarial Robustness of Language Models

Explore similar work

IndicSafeEval: Safety Robustness of Large Language Models under Multilingual Persuasive Jailbreak Attacks

IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asia

SurakshaEval: An Indic Safety Benchmark for Multilingual LLMs