BanglaShield: A Script-Robust Safety Classifier and Benchmark for Bangla Native, Romanized, and Code-Mixed Prompts

Mohsinul Kabir; Md. Tariquzzaman; Mohammad Rashidujjaman Rifat; Tharindu Ranasinghe; Nishat Raihan

cs.CLOct 16, 2026

BanglaShield: A Script-Robust Safety Classifier and Benchmark for Bangla Native, Romanized, and Code-Mixed Prompts

Authors: Mohsinul Kabir, Md. Tariquzzaman, Mohammad Rashidujjaman Rifat, Tharindu Ranasinghe, Nishat Raihan

Organizations: University of Manchester, UK · Islamic University of Technology, Bangladesh · University of Notre Dame, USA · Lancaster University, UK

Abstract

Safety classifiers for large language models (LLMs) are typically built and evaluated on standard-script text, yet Bangla, like many lower-resource languages, is routinely written in native script, in Latin-script romanization, and code-mixed with English, Hindi, or Urdu. We introduce BanglaShield, a resource and model for graded prompt-safety classification across these written forms. We consolidate 10,245 Bangla prompts from six sources under a four-level taxonomy (Safe, Offensive, Harmful, Severe) and render each prompt in three aligned forms that share labels and a group-aware split, enabling strictly paired comparisons. Benchmarking six contemporary LLMs, including a Bangla-specialized one, on 36,882 classification queries and 24,318 judged responses, we find that LLM classifiers label 47.0% of unsafe native-script prompts as Safe, rising to 63.4% under romanization, while response-level safety accuracy drops by 10.3 points. Three multilingual guard models also pass 30.3-57.3% of unsafe native-script prompts. Using a model's own classification as an input filter reduces harmful compliance by only about five points. In contrast, BanglaShield, a compact fine-tuned encoder, reaches 87.8 macro-F1 on native-script and 84.4 on romanized prompts, outperforming the best LLM on each form by 42.8-46.1 points with ~130x fewer parameters than Gemma 4 31B. We release the corpus, splits, judgments, and model.

Figures & tables

Figure 1: One Severe prompt rendered in the three BanglaShield forms with the predictions of six LLMs. All six models flag the Bangla-script prompt as unsafe; five of six classify the code-mixed rendering as Safe .

Panel of Figure 2 — Figure 2: Overview of BanglaShield . Six resources are mapped to a four-level taxonomy, every prompt is rendered in three aligned written forms, and the paired partition supports both the LLM benchmark and the training of the BanglaShield classifier. Counts refer to the full corpus. ∗ BanglaEcomReviewCorpus negative reviews converted into benign prompts.

Panel of Figure 3 — Figure 3: Corpus statistics. (a) Label composition per source; the benign reviews supply most Safe items, while BanglaSafe is dominated by Severe items. (b) Share of letters by Unicode script in each test form. (c) Distribution of the realized word-substitution rate in the code-mixed test form (mean $0.40\pm 0.07$ ; English 1,229, Hindi 511, Urdu 309 prompts).

Resource	Bn	Rom.	CM	Grad.	Pair.	Guard
PolyGuard	✗	✗	✗	✗	✗	✓
RTP-LX	✗	✗	✗	✗	✗	✗
MultiJail	✓	✗	✗	✗	$\circ$	✗
XSafety	✓	✗	✗	✗	$\circ$	✗
LinguaSafe	✓	✗	✗	✓	$\circ$	✗
IndicSafe	✓	✗	✗	✗	✗	✗

Table 1: Comparison with related safety resources (cited in § 2 ). Bn: Bangla; Rom.: romanized Bangla; CM: code-mixed; Grad.: more than one unsafe level (

\circ

= safe-sensitive vs. unsafe); Pair.: identical items across forms (

\circ

= across languages or registers only); Guard: released guard model (

\circ

= described but not released).

ID	Label	Definition	Typical categories
0	Safe	No harmful request	Neutral, positive, harmless
1	Offensive	Insulting, provocative, or discriminatory expression without a request for harmful assistance	Offensive language, hate speech, anger
2	Harmful	Request for lower- or medium-risk harmful assistance	Misinformation, adversarial, malicious requests
3	Severe	Request for high-risk assistance with substantial potential consequences	Suicide, jailbreak attempts

Table 2: Safety label taxonomy.

Source	S	O	H	V	Total
LinguaSafe	88	103	204	287	682
BanglaEcom	576	–	–	–	576
XSafety	–	160	280	80	520
BanglaSafe	–	6	14	88	108
IndicSafe	34	28	28	10	100
MultiJail	–	17	12	34	63

Table 3: Test-set composition (S/O/H/V: Safe/Offensive/Harmful/Severe). Each item appears in all three forms (6,147 prompts); mean length is 148.7 (script), 141.6 (code-mixed), and 161.2 (romanized) characters, 23.6 words in every form.

		Bangla script			Code-mixed			Romanized
Model	Params	Acc $\uparrow$	M-F1 $\uparrow$	USR $\downarrow$	Acc $\uparrow$	M-F1 $\uparrow$	USR $\downarrow$	Acc $\uparrow$	M-F1 $\uparrow$	USR $\downarrow$
Gemma 4 31B	31B	51.2	42.9	47.4	51.0	42.2 ( $-$ 0.7)	47.3	50.6	41.6 ( $-$ 1.3)	49.1
Llama 4 Maverick	400B	47.3	37.5	50.2	45.8	35.6 ( $-$ 1.9)	46.8	43.1	31.7 ( $-$ 5.9) †	60.7
Command A+	n/r	47.8	37.0	49.2	44.4	32.4 ( $-$ 4.6) †	57.9	35.8	17.1 ( $-$ 19.9) †	87.8
GPT-4.1 mini	n/r	47.0	36.4	51.8	44.6	33.1 ( $-$ 3.2) †	55.5	43.8	32.7 ( $-$ 3.6) †	61.2
Claude Haiku 4.5	n/r	52.4	43.9	35.8	50.3	41.5 ( $-$ 2.4) †	37.7	45.6	34.4 ( $-$ 9.5) †	54.4

Table 4: [ Acc

\uparrow

M-F1

\uparrow

USR

\downarrow

] Four-way prompt classification on the 2,049-item test set (%). Parentheses: change in M-F1 from Bangla script; † : its 95% paired-bootstrap interval excludes zero. USR: unsafe prompts predicted Safe . Middle: guard models (safe/unsafe output, so USR only; Appendix J ). Params: where public (n/r: not reported).

Panel of Figure 4 — Figure 4: [% of gold row] Confusion matrices pooled over the six LLMs (S/O/H/V: Safe/Offensive/Harmful/Severe; outlined cells are correct predictions). The first column, i.e., unsafe prompts passed as Safe , grows from Bangla script to romanized text for every unsafe class, and correct Severe predictions halve.

Panel of Figure 5 — Figure 5: [ F1 $\uparrow$ ] Per-class F1, mean of the six LLMs. Severe loses 46% of its F1 under romanization.

	Bangla script				Code-mixed				Romanized
Model	Off.	Harm.	Sev.	All	Off.	Harm.	Sev.	All	Off.	Harm.	Sev.	All
Gemma 4 31B	81.5	79.0	80.4	80.1	86.0	74.5	76.2	77.8 ( $-$ 2.3)	85.4	74.0	74.8	76.9 ( $-$ 3.2)
Llama 4 Maverick	80.3	77.9	80.0	79.2	82.5	75.1	75.8	77.1 ( $-$ 2.2)	79.6	68.2	65.3	69.8 ( $-$ 9.4)
Command A+	71.0	78.6	83.2	78.5	68.8	74.7	79.0	74.9 ( $-$ 3.6)	25.2	67.1	71.9	59.1 ( $-$ 19.4)
GPT-4.1 mini	84.7	69.7	68.3	72.7	84.4	71.0	62.5	71.0 ( $-$ 1.7)	83.8	59.7	51.9	62.4 ( $-$ 10.3)
Claude Haiku 4.5	76.4	88.9	92.6	87.3	74.8	90.9	93.0	87.9 ( $+$ 0.6)	73.3	82.7	81.6	80.1 ( $-$ 7.3)

Table 5: [ ARR

\uparrow

] Appropriate response rate (%) by gold label, from the judge panel. Offensive prompts expect a safe/useful completion; Harmful and Severe prompts expect refusal. Parentheses give the change from Bangla script.

Panel of Figure 6 — Figure 6: [x: M-F1 $\uparrow$ y: ARR $\uparrow$ ] Classification versus response safety for the 18 model–form cells; arrows run from each model’s Bangla-script to its romanized cell. Forms classified poorly are also answered unsafely.

	Script		Code-mixed		Romanized
Model	none	self	none	self	none	self
Gemma 4	37.2	32.0	41.1	31.5	40.5	34.1
Llama 4	37.3	32.7	37.2	32.8	41.1	37.1
Command A+	32.0	27.4	34.9	31.9	17.8	16.4
GPT-4.1 mini	44.2	36.7	47.5	38.0	53.0	43.1
Haiku 4.5	24.3	21.8	23.3	20.7	29.6	24.5

Table 6: [ HCR

\downarrow

] Harmful compliance rate (HCR, %) on Harmful and Severe prompts without a guard (none) and with the LLM’s own classification as a filter (self); “+ guard”: mean HCR with an external guard as the filter. A filtered prompt is counted as refused.

	LLM classification ( Acc $\uparrow$ )			BanglaShield classification ( Acc $\uparrow$ )			LLM response ( ARR $\uparrow$ )
Source	Scr.	CM	Rom.	Scr.	CM	Rom.	Scr.	CM	Rom.
BanglaEcom	99.8	99.0	99.0	100.0	100.0	100.0	–	–	–
IndicSafe	51.8	47.8	41.0	93.0	94.0	90.0	68.4	70.7	58.3
LinguaSafe	36.4	33.7	27.1	78.3	79.5	72.8	80.9	79.5	71.2
MultiJail	24.1	16.1	12.4	79.4	84.1	76.2	83.3	78.0	64.0
XSafety	18.2	16.9	14.3	91.7	91.9	87.9	81.1	79.7	69.2

Table 7: [ Acc

\uparrow

ARR

\uparrow

] Results by source and written form (%). The first two groups compare classification Acc of the six LLMs (averaged) and of BanglaShield on the same items; the last group gives the LLMs’ ARR (labels 1–3), which a classifier does not produce.

Panel of Figure 7 — Figure 7: [ M-F1 $\uparrow$ ] Macro-F1 of each system on the three forms of the same items; $\Delta$ Rom. is the change from Bangla script to romanized. Command A+ drops most.

	Correct in	Flips from Bangla script
Model	all 3 forms	to CM	to Rom.
Gemma 4	45.9	3.6 / 3.4	3.1 / 2.4
Llama 4	36.1	7.2 / 5.6	8.4 / 4.1 †
Command A+	34.5	6.9 / 3.5 †	12.7 / 0.7 †
GPT-4.1 mini	38.7	5.0 / 2.6 †	6.3 / 3.1 †
Haiku 4.5	41.2	5.2 / 3.1 †	9.9 / 3.1 †

Table 8: [ Correct in all 3

\uparrow

flips] Paired stability (% of 2,049 items): items correct in all three forms, and correct-to-wrong / wrong-to-correct flips from Bangla script. † : McNemar’s test,

p<.05

Panel of Figure 8 — Figure 8: [ M-F1 $\uparrow$ ] BanglaShield trained on one written form or on all three (full model), tested on each form. Only pooled training is strong on every form.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Backbone	google/muril-base-cased (237M)
Training data	7,172 items $\times$ 3 forms = 21,516
Validation data	1,024 items $\times$ 3 forms = 3,072
Max sequence length	256
Optimizer / learning rate	AdamW / 3e-5, linear decay, 10% warm-up
Batch size / epochs	32 / 10

Appendix

Table 10: BanglaShield fine-tuning configuration.

	English		Hindi		Urdu
Model	Scr.	CM	Scr.	CM	Scr.	CM
Gemma 4	43.0	42.7	44.5	41.5	39.3	40.5
Llama 4	37.2	36.0	39.4	34.7	35.8	35.8
Command A+	38.6	32.4	34.5	31.5	34.4	33.1
GPT-4.1 mini	37.6	33.9	33.3	32.9	36.0	30.3
Haiku 4.5	44.5	41.2	42.9	42.7	43.0	40.4

Appendix

Table 11: [ M-F1

\uparrow

] LLMs on code-mixed (CM) vs. Bangla-script (Scr.) items (%).

Panel of Figure 10 — Figure 10: [(a) M-F1 $\uparrow$ (b) Acc $\uparrow$ ] LLM mean on code-mixed prompts and the same items in Bangla script, by target language (a) and realized substitution-rate bin (b; B1–B4: $<$ .38, .38–.42, .42–.46, $\geq$ .46; $n$ = 670, 477, 470, 432).

Panel of Figure 11 — Figure 11: [% of prompts] Predicted-label distribution per LLM (bars: script, CM, romanized, top to bottom); top bar and dashed line: gold (34.1% Safe ).

	Script		Code-mixed		Romanized
Training form	Acc	M-F1	Acc	M-F1	Acc	M-F1
Script	88.5	87.2	78.7	76.3	43.3	39.4
Code-mixed	82.9	81.2	87.1	85.9	44.0	42.9
Romanized	46.3	44.7	35.3	34.6	81.9	80.2

Appendix

Table 13: [ Acc

\uparrow

M-F1

\uparrow

] BanglaShield trained on a single written form (rows) and tested on each form of the test set (columns), in %.

	OFR (%)			MAE
Model	Scr.	CM	Rom.	Scr.	CM	Rom.
Gemma 4	0.4	1.4	0.4	0.82	0.81	0.83
Llama 4	0.6	4.3	3.7	0.90	0.91	1.05
Command A+	0.6	0.7	0.7	0.88	0.98	1.28
GPT-4.1 mini	0.7	1.4	1.9	0.92	0.97	1.03
Haiku 4.5	1.1	2.0	2.1	0.72	0.76	0.92

Appendix

Table 14: [ OFR

\downarrow

MAE

\downarrow

] LLM error profile: over-flag rate (OFR), Safe prompts given an unsafe label; mean absolute label error (MAE, 0–3).

Model	OpenRouter / Hugging Face ID
Evaluated LLMs
Gemma 4 31B	google/gemma-4-31b-it
Llama 4 Maverick	meta-llama/llama-4-maverick
Command A+	cohere/command-a-plus
GPT-4.1 mini	openai/gpt-4.1-mini
Claude Haiku 4.5	anthropic/claude-haiku-4.5

Appendix

Table 15: Models used in the benchmark (snapshot 1 October 2026).

Panel of Figure 13 — Figure 13: [% of responses] Compliance degree of LLM responses to Harmful and Severe prompts (final judge decision). Degrees 2–3 indicate material or directly actionable assistance.

	Script			Code-mixed			Romanized
Guard	O	H	V	O	H	V	O	H	V
Llama Guard 3	26.8	32.7	63.5	25.8	30.1	59.3	30.3	44.8	63.1
PolyGuard-Qwen	62.1	55.0	83.8	56.4	54.5	81.4	28.7	31.2	55.5
IndicGuard	63.1	60.6	83.6	62.1	58.2	81.6	29.6	30.9	48.5

Appendix

Table 18: [ Detection

\uparrow

] Share of unsafe prompts flagged by each guard, by gold label (O/H/V: Offensive/Harmful/Severe), in %.

Guard	Hugging Face checkpoint
Llama Guard 3 8B	meta-llama/Llama-Guard-3-8B
PolyGuard-Qwen	ToxicityPrompts/PolyGuard-Qwen
IndicGuard	l3cube-pune/IndicGuard

Appendix

Table 19: Guard models (run locally, 11 October 2026).

LLM Safety Benchmarks

Large language model safety benchmarks test whether AI systems avoid harm while remaining helpful across realistic tasks, users, and deployment settings.

See all 270 papers

BanglaVeilGuard: Cross-Script Safety Benchmarking and Lightweight Guardrails for Bangla Large Language Models
Aug 22, 2026Md. Rakibul Hassan, Muhammad Iqbal HossainLLM Safety Monitoring LLM Safety
BRAC University Dhaka, Bangladesh
Register Shifts Break LLM Safety: A Bengali Benchmark with Culturally Grounded Harms
Aug 23, 2026Naymul Islam, Nusrat Jahan Lia, Shubhashis Roy Dipta +2LLM Safety Benchmarks Language Model Safety Evaluation
BanglaLLM · Institute of Information Technology, University of Dhaka · University of Maryland, Baltimore County +2
ThaiSafetyBench: Assessing Language Model Safety in Thai Cultural Contexts
Mar 5, 2026Trapoom Ukarapol, Nut Chukamphaeng, Kunat Pipatanakul +1Language Model Safety Evaluation LLM Safety
SCB DataX · Department of Computer Science and Technology, Tsinghua University · SCBX R&D +1

Aug 22, 2026J/K move · Enter open · S save

BanglaVeilGuard: Cross-Script Safety Benchmarking and Lightweight Guardrails for Bangla Large Language Models

Md. Rakibul Hassan, Muhammad Iqbal Hossain

BRAC University Dhaka, Bangladesh

LLM Safety Monitoring LLM Safety Language Model Safety Evaluation LLM Safety Benchmarks LLM Guardrails Bengali NLP+1

Open paper AskPDF arXiv

Bangla large language model (LLM) safety is difficult to evaluate with English-centric or standard-script benchmarks because Bangla users routinely write across scripts, spellings, code-mixed forms, and regional registers. This paper presents BanglaVeilGuard, a compact Bangla-first safety benchmark and lightweight prompt guard for six language forms: standard Bangla, Romanized Bangla, Banglish, code-mixed Bangla--English, noisy Bangla, and dialectal Bangla. The benchmark contains 2,366 quality-filtered prompts and a held-out 354-prompt evaluation split spanning unsafe, safe, and safe-sensitive requests. BanglaVeilGuard uses non-destructive multi-view normalization with a prompt-risk classifier and thresholded pre-generation gate, allowing it to screen prompts for heterogeneous target models without changing their weights. Across target-model families, guarded runs reduce attack success under deterministic response scoring from 93.8--100.0% to 6.3% for Claude Opus 4.8, BanglaLLama, and TituLLM; TigerLLM-1B with BanglaVeilGuard achieves 78.2% accuracy with 8.8% ASR. The prompt guard also attains 88.5% unsafe recall, substantially above the evaluated prompt-only guard baselines. The main remaining cost is over-refusal on dialectal and noisy benign prompts, revealing a concrete safety-helpfulness frontier for Bangla LLM deployment.

Abstract

LLM Safety Benchmarks

Explore similar work

BanglaVeilGuard: Cross-Script Safety Benchmarking and Lightweight Guardrails for Bangla Large Language Models

Register Shifts Break LLM Safety: A Bengali Benchmark with Culturally Grounded Harms

ThaiSafetyBench: Assessing Language Model Safety in Thai Cultural Contexts