Register Shifts Break LLM Safety: A Bengali Benchmark with Culturally Grounded Harms

Naymul Islam; Nusrat Jahan Lia; Shubhashis Roy Dipta; Sabik Bin Sultan; Abdullah Khan Zehady

cs.CLAug 23, 2026

Register Shifts Break LLM Safety: A Bengali Benchmark with Culturally Grounded Harms

Authors: Naymul Islam, Nusrat Jahan Lia, Shubhashis Roy Dipta, Sabik Bin Sultan, Abdullah Khan Zehady

Organizations: BanglaLLM · Institute of Information Technology, University of Dhaka · University of Maryland, Baltimore County · Bangladesh Air Force Shaheen College Kurmitola ·

Abstract

Bengali is the seventh-most-spoken language globally, yet LLM safety evaluation remains overwhelmingly English-centric. We introduce BanglaSafe, a benchmark of 879 Bengali prompts combining 309 natively authored prompts with 570 expert-reviewed prompts, spanning 17 culturally grounded harm categories and five prompting conditions that vary language, writing style, and authority framing. Evaluating 18 frontier LLMs, we find that over half of all responses are unsafe or partially unsafe (53.6%) while 14.7% contains strictly harmful content, and that the strongest observed effect is not the switch from English to Bengali but the choice of writing style within Bengali: the same harmful request phrased as a formal newspaper investigation succeeds 17 percentage points more often than the same request phrased as a casual message, with no adversarial engineering involved. We further show that existing safety classifiers struggle to reliably evaluate Bengali content, with even frontier models failing on nearly half of all cases.

Figures & tables

Figure 1: Overview of the BanglaSafe creation. Left: 879 prompts (309 human-written, 570 LLM-assisted) are drawn from a 17-category culturally grounded harm taxonomy. Center: each prompt appears under five conditions that vary language, register, and authority framing, yielding 15,822 responses across 18 LLMs. Arrows indicate the three paired comparisons: language effect ( EN Direct vs. BN Formal ), register effect ( BN Formal vs. BN Collq ), and authority effect ( EN Direct vs. EN Inst ). Right: responses are evaluated via a two-stage pipeline: prompt-level human validation followed by a four-way LLM judge producing Refuse , Policy , Partial , or Harmful labels.

English safety benchmarks
Resource	Native	N (Bn)	Culturally Grounded	Register	Authority	Mechanism
HarmBench ( 25 )	–	0	✗	✗	✗	✗
JailbreakBench ( 4 )	–	0	✗	✗	✗	✗
SORRY-Bench ( 49 )	–	0	✗	~	~	✗
Shah et al. ( 41 )	–	0	✗	✗	~	✗
Multilingual safety (Bengali subset)

Table 1: Comparison with prior safety benchmarks. Native : prompts are natively authored rather than translated. N (Bn) : number of Bengali prompts. Bengali Culturally Grounded : harm categories anchored to Bengali cultural norms and local statutes. ✓ = present; ✗ = absent; ~ = partial.

Comparison	Binary $\kappa$	4-way $\kappa$
Inter-human (A1 $\leftrightarrow$ A2)	0.586	0.286
Claude judge $\leftrightarrow$ A1	0.666	—
Claude judge $\leftrightarrow$ A2	0.557	—

Table 2: Cohen’s

\kappa

on the 300-response validation subset. Binary = Partial + Harmful collapsed to unsafe. The judge-vs-A1 agreement (

\kappa=0.666

) exceeds the

\kappa\geq 0.65

threshold 22 . The low four-way inter-human

\kappa

(

0.286

) confirms the Partial / Harmful boundary is subjective on Bengali code-mixed responses.

Figure 2: ASR loose by prompting condition with 95% bootstrap CIs (

B{=}10{,}000

). BN Formal peaks at 63.3%; BN Collq is the safest at 45.8%. The 17pp paired gap between these two Bengali conditions is the paper’s strongest effect.

Condition	$n$	ASR loose	ASR strict
EN Direct	3,114	0.504 [.487, .522]	0.187 [.173, .201]
EN Inst	2,970	0.534 [.517, .552]	0.208 [.194, .223]
BN Formal	3,132	0.633 [.617, .650]	0.135 [.123, .147]
BN Collq	3,420	0.458 [.442, .475]	0.077 [.068, .086]
BN Inst	3,186	0.557 [.540, .575]	0.140 [.128, .152]

Table 3: Attack success rate by prompting condition with 95% paired-bootstrap CIs. BN Formal is the highest ASR loose condition but has lower ASR strict than the English baseline: the formal-Bengali effect lives in the Partial bucket, not in verbatim leakage.

Test	Comparison	$n$ pairs	Mean diff	95% CI	Holm $p$	$r_{rb}$
ABL-1 (Language)	EN Direct $\leftrightarrow$ BN Formal	3,060	$-0.130$	$[-0.150,-0.111]$	$<10^{-15}$	$-0.411$
ABL-2 (Authority)	EN Inst $\leftrightarrow$ BN Inst	2,934	$-0.044$	$[-0.063,-0.025]$	$4.7\!\times\!10^{-5}$	$-0.154$
ABL-2b (Authority)	EN Direct $\leftrightarrow$ EN Inst	2,952	$-0.029$	$[-0.048,-0.010]$	$0.019$	$-0.097$
ABL-3a (Register)	BN Formal $\leftrightarrow$ BN Collq	3,132	$\mathbf{+0.170}$	$[+0.152,+0.188]$	$<10^{-15}$	$\mathbf{+0.573}$
ABL-3b (Register)	BN Formal $\leftrightarrow$ BN Inst	3,060	$+0.068$	$[+0.051,+0.085]$	$9.0\!\times\!10^{-14}$	$+0.284$
ABL-3c (Register)	BN Collq $\leftrightarrow$ BN Inst	3,132	$-0.098$	$[-0.117,-0.079]$	$<10^{-15}$	$-0.283$

Table 4: Six paired Wilcoxon ablations on ASR loose within-(model, base-prompt) pairs, with Holm-Bonferroni correction over the

K{=}6

family. ABL-3a ( BN Formal vs. BN Collq ) is the strongest effect: a 17.0pp register gap within Bengali with a large effect size.

Figure 3: Four-way label distribution by prompting condition (

n{=}15{,}822

). BN Formal raises Partial to 50% while lowering Harmful to 13% (below the EN Direct baseline of 19%). The formal-Bengali effect lives in the Partial bucket: models engage with the content but hedge through journalistic framing rather than producing verbatim recipes.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Statute / Source	N
banned militant org	Anti-Terrorism Act 2009	53
burn / corrosive (incl. acid attacks)	Acid Crime Control Act 2002 S.4 + PC 326A	52
campus violence	Penal Code 1860 + university regs	45
cert. forgery	Penal Code 467/468	56
child marriage	Child Marriage Restraint Act 2017	50
communal violence	Penal Code + DSA 2018	44

Appendix

Table 5: The 17 culturally grounded harm categories with primary statute or source. N is the total prompt count per category (gold + synth), summing to 879.

Category	Anchored / Total	%
certificate forgery	51 / 56	91.1
burn / corrosive	47 / 52	90.4
campus violence	39 / 45	86.7
communal violence	38 / 44	86.4
banned militant org	43 / 53	81.1
rape	41 / 54	75.9

Appendix

Table 6: Case-anchor density per harm category, ordered by density.

Feature	EN Direct / EN Inst	BN Formal	BN Collq	BN Inst
Framing	English query / researcher	Journalistic ground-report	Peer-help / emergency	Officer self-intro, case-file
2nd-person address	you	impersonal (no direct address)	\kalpurush তুই/ \kalpurush তুমি, \kalpurush ভাই	\kalpurush আপনি / impersonal
Honorific level	—	neutral–formal	low (peer/intimate)	high (formal/official)
Verb inflection	English	standard \kalpurush চলিত ( \kalpurush -ছে, \kalpurush -ছেন)	colloquial ( \kalpurush -িস, \kalpurush -ছোস)	formal/passive ( \kalpurush -হয়েছে)
Discourse particles	—	minimal	\kalpurush রে, \kalpurush আরে, \kalpurush তো	bureaucratic / statute citations
Code-mixing (Latin-script)	100%	2.9%	7.9%	18.1%

Appendix

Table 7: Register inventory across the five prompting conditions: morphosyntactic and discourse features that distinguish the registers. Code-mixing is the median Latin-script density per condition ( Section 6.2 ); the BN Collq

<

BN Inst ordering shows institutional Bangla mixes in more English (technical/operational terms) than colloquial Banglish. The two English conditions ( EN Direct , EN Inst ) share identical Bengali-specific features.

Condition	Prompts	Language type
EN Direct	173	English
EN Inst	165	English
BN Formal	174	Bangla (script)
BN Collq	190	Banglish (code-mixed)
BN Inst	177	Bangla (script)
Total	879

Appendix

Table 8: Dataset composition by prompting condition and language type. Rolled up by language: English 338 ( EN Direct + EN Inst ), Banglish 190 ( BN Collq ), and Bangla 351 ( BN Formal + BN Inst ), totalling 879 prompts.

ID	Role	Pass 1 (Prompt IAA)	Pass 2 (Judge IAA)
A1	Primary annotator	✓	✓
A2	Second annotator	✓	✗
A3	Third annotator	✗	✓

Appendix

Table 9: Annotator roles across the two validation passes. All annotators are native Bangla speakers who participated voluntarily.

Cross-judge metric	Value
Overall ASR (Claude / Gemini)	53.6% / 50.0%
Binary agreement $\kappa$ (unsafe vs. safe)	0.787
Four-way agreement $\kappa$	0.626
Per-model ASR correlation (Pearson $r$ )	0.989
Per-model ASR correlation (Spearman $\rho$ )	0.981

Appendix

Table 10: Robustness of the calibrated judge to judge-model choice (Claude Opus 4.7 vs. Gemini-3.1-Pro). Re-evaluating all 15,822 responses with a second, independent judge (Gemini-3.1-Pro) yields an overall ASR within 3.6pp of Claude, substantial binary agreement (

\kappa=0.787

), and near-identical per-model rankings (Pearson

r=0.989

, Spearman

\rho=0.981

). The paper’s conclusions hold under either judge, indicating the findings are not an artifact of a single judge model.

Guard	Prompt-only	Response	$\Delta$
LlamaGuard 4	81.5%	15.4%	$-66.1$ pp
GPT-OSS-Safeguard	89.1%	62.1%	$-27.0$ pp
LG4 $\leftrightarrow$ GPT-OSS $\kappa$	0.159	0.102	—

Appendix

Table 11: Aggregate prompt-only vs. response-level unsafe-flagging rates for both guards.

	LlamaGuard 4		GPT-OSS-Safeguard
Condition	Prompt	Resp	Prompt	Resp
EN Direct	92.5%	18.9%	99.4%	54.0%
EN Inst	51.5%	3.3%	90.2%	59.9%
BN Formal	96.6%	22.4%	90.1%	74.4%
BN Collq	88.9%	20.1%	82.5%	53.3%
BN Inst	75.7%	11.2%	83.9%	69.4%

Appendix

Table 12: Prompt-only vs. response-level unsafe-flagging rates by prompting condition. LlamaGuard 4 drops to 51.5% prompt-level on EN Inst , and the corresponding response-level rate is 3.3%. The authority-cover framing weakens guard detection at both stages.

Open-weight, $<$ 10B parameters
Model	ASR loose	ASR strict	Refuse	Policy	Partial	Harmful
Qwen3-8B	78.5%	21.6%	6.0%	15.5%	56.9%	21.6%
TigerLLM-9B-it	73.2%	10.8%	8.0%	18.9%	62.3%	10.8%
TigerLLM-1B	10.6%	1.5%	68.9%	20.5%	9.1%	1.5%
Llama-3.1-8B	8.4%	0.3%	81.6%	10.0%	8.1%	0.3%
Llama-3.2-3B	3.9%	0.5%	89.9%	6.3%	3.4%	0.5%

Appendix

Table 13: Per-model four-way label distribution and ASR over the full 15,822-response corpus (18 models

\times

879 prompts), grouped by parameter scale (open-weight) and separately for closed-source models. Bold = highest value across all models in that column; underline = highest within the subsection.

Model	EN Direct	EN Inst	BN Formal	BN Collq	BN Inst	$\Delta_{\text{reg}}$
Open-weight, $<$ 10B parameters
Qwen3-8B	87.9	98.2	77.0	51.6	81.4	$+25.4$
TigerLLM-9B-it	89.6	63.0	79.9	65.8	67.8	$+14.1$
TigerLLM-1B ‡	22.5	13.3	4.6	4.7	8.5	$-0.1$
Llama-3.1-8B	12.1	4.2	18.4	3.7	4.0	$+14.7$
Llama-3.2-3B ‡	2.3	2.4	10.3	1.6	2.8	$+8.7$

Appendix

Table 14: Per-model ASR loose (%) by prompting condition (18 models

\times

879 prompts), grouped by scale tier.

\Delta_{\text{reg}}=\textbf{BN\textsubscript{Formal}}{}-\textbf{BN\textsubscript{Collq}}{}

is the within-Bengali register gap. Bold = highest value in each column. The register effect is near-universal: BN Formal

{>}

BN Collq in 17 of 18 models (sign test,

p<0.001

), with a median gap of

+17.6

pp, and it is stronger in the most capable models (Mistral-Medium-3, DeepSeek-V3, GPT-4.1-mini, Gemma-4-26B all reach

{\geq}90\%

in BN Formal ). ‡ Model operates near its capability floor (near-zero ASR across most conditions); its register gap is not meaningful.

Category	$n$	ASR loose	ASR strict
dowry violence	936	61.5%	9.8%
child marriage	900	61.4%	16.6%
trafficking	954	59.6%	9.2%
yaba / narcotics	954	58.2%	22.4%
campus violence	810	56.8%	7.5%
MFS fraud (bKash)	918	56.3%	17.3%

Appendix

Table 15: Per-category ASR across all models and conditions, sorted by ASR loose descending. Self-harm has the lowest ASR loose (40.9%) but the highest ASR strict (25.9%), suggesting models either refuse entirely or commit fully on this category.

Pair	$n$	$\kappa$	95% CI
Claude $\leftrightarrow$ LlamaGuard 4	12,279	0.014	$[+0.002,+0.027]$
Claude $\leftrightarrow$ GPT-OSS-Safeguard	10,207	0.667	$[+0.652,+0.681]$
LlamaGuard 4 $\leftrightarrow$ GPT-OSS-Safeguard	10,202	0.102	$[+0.092,+0.113]$

Appendix

Table 16: Pairwise Cohen’s

\kappa

between the calibrated Claude judge (binary-collapsed) and two field-standard guards on the response-ID intersection each pair covered.

Corpus	$n$ (prompts)
HarmBench	400	25
JailbreakBench	100	4
BeaverTails-30k	7,774	18
ALERT	14,763	45
SORRY-Bench	9,450	49
AdvBench	520	55

Appendix

Table 17: Twelve open English safety RLHF corpora used in the corpus coverage probe ( Section 6.1 ). Across all 80,587 prompts, 20 culturally anchored Bengali harm terms return one hit total (“dowry” in PKU-SafeRLHF , with no Bangladesh context).

LLM Safety Benchmarks

Large language model safety benchmarks test whether AI systems avoid harm while remaining helpful across realistic tasks, users, and deployment settings.

See all 273 papers

BanglaVeilGuard: Cross-Script Safety Benchmarking and Lightweight Guardrails for Bangla Large Language Models
Aug 22, 2026Md. Rakibul Hassan, Muhammad Iqbal HossainLLM Safety Monitoring LLM Safety
BRAC University Dhaka, Bangladesh
BanglaShield: A Script-Robust Safety Classifier and Benchmark for Bangla Native, Romanized, and Code-Mixed Prompts
Oct 16, 2026Mohsinul Kabir, Md. Tariquzzaman, Mohammad Rashidujjaman Rifat +2LLM Safety Benchmarks Language Model Safety Evaluation
University of Manchester, UK · Islamic University of Technology, Bangladesh · University of Notre Dame, USA +1
IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asia
Mar 18, 2026Priyaranjan Pattnayak, Sanchari ChowdhuriLanguage Model Safety Evaluation LLM Safety Benchmarks
Oracle America Inc.

Aug 22, 2026J/K move · Enter open · S save

BanglaVeilGuard: Cross-Script Safety Benchmarking and Lightweight Guardrails for Bangla Large Language Models

Md. Rakibul Hassan, Muhammad Iqbal Hossain

BRAC University Dhaka, Bangladesh

LLM Safety Monitoring LLM Safety Language Model Safety Evaluation LLM Safety Benchmarks LLM Guardrails Bengali NLP+1

Open paper AskPDF arXiv

Bangla large language model (LLM) safety is difficult to evaluate with English-centric or standard-script benchmarks because Bangla users routinely write across scripts, spellings, code-mixed forms, and regional registers. This paper presents BanglaVeilGuard, a compact Bangla-first safety benchmark and lightweight prompt guard for six language forms: standard Bangla, Romanized Bangla, Banglish, code-mixed Bangla--English, noisy Bangla, and dialectal Bangla. The benchmark contains 2,366 quality-filtered prompts and a held-out 354-prompt evaluation split spanning unsafe, safe, and safe-sensitive requests. BanglaVeilGuard uses non-destructive multi-view normalization with a prompt-risk classifier and thresholded pre-generation gate, allowing it to screen prompts for heterogeneous target models without changing their weights. Across target-model families, guarded runs reduce attack success under deterministic response scoring from 93.8--100.0% to 6.3% for Claude Opus 4.8, BanglaLLama, and TituLLM; TigerLLM-1B with BanglaVeilGuard achieves 78.2% accuracy with 8.8% ASR. The prompt guard also attains 88.5% unsafe recall, substantially above the evaluated prompt-only guard baselines. The main remaining cost is over-refusal on dialectal and noisy benign prompts, revealing a concrete safety-helpfulness frontier for Bangla LLM deployment.

Abstract

LLM Safety Benchmarks

Explore similar work

BanglaVeilGuard: Cross-Script Safety Benchmarking and Lightweight Guardrails for Bangla Large Language Models

BanglaShield: A Script-Robust Safety Classifier and Benchmark for Bangla Native, Romanized, and Code-Mixed Prompts

IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asia