BURMESE-SAN: Burmese NLP Benchmark for Evaluating Large Language Models

Thura Aung; Jann Railey Montalan; Jian Gang Ngui; Peerat Limkonchotiwat

cs.CLFeb 21, 2026

BURMESE-SAN: Burmese NLP Benchmark for Evaluating Large Language Models

Authors: Thura Aung, Jann Railey Montalan, Jian Gang Ngui, Peerat Limkonchotiwat

Organizations: King Mongkut’s Institute of Technology Ladkrabang · AI Singapore · National University of Singapore

Abstract

We introduce BURMESE-SAN, the first holistic benchmark that systematically evaluates large language models (LLMs) for Burmese across three core NLP competencies: understanding (NLU), reasoning (NLR), and generation (NLG). BURMESE-SAN consolidates seven subtasks spanning these competencies, including Question Answering, Sentiment Analysis, Toxicity Detection, Causal Reasoning, Natural Language Inference, Abstractive Summarization, and Machine Translation, several of which were previously unavailable for Burmese. The benchmark is constructed through a rigorous native-speaker-driven process to ensure linguistic naturalness, fluency, and cultural authenticity while minimizing translation-induced artifacts. We conduct a large-scale evaluation of both open-weight and commercial LLMs to examine challenges in Burmese modeling arising from limited pretraining coverage, rich morphology, and syntactic variation. Our results show that Burmese performance depends more on architectural design, language representation, and instruction tuning than on model scale alone. In particular, Southeast Asia regional fine-tuning and newer model generations yield substantial gains. Finally, we release BURMESE-SAN as a public leaderboard to support systematic evaluation and sustained progress in Burmese and other low-resource languages. https://leaderboard.sea-lion.ai/detailed/MY

Figures & tables

Figure 1: BURMESE-SAN Benchmark (Left) and Dataset Curation Process for the benchmark (Right). BURMESE-SAN is a benchmark that holistically evaluates LLM performance across a wide range of Burmese language tasks. The evaluation is based on native Burmese text, with prompts written in formal Burmese to ensure clarity and grammatical correctness.

Comp.	Task	Dataset	Target 1	Language	Source 2	Adaptation	Our contribution
NLU	QA	Belebele Bandarkar et al. (2024)	S (4)	Burmese	English-adapted	Native re-translation	✓
	SA	GKLMIP-mya Jiang et al. (2021)	L (3)	Burmese	Natively-sourced	Normalization 3	✓
	TD	myHateSpeech Kyaw et al. (2024)	L (9)	Burmese	Natively-sourced	No Adaptation
NLR	CR	Balanced COPA Kavumba et al. (2019)	L (2)	English	English-sourced	Native translation	✓
	NLI	myXNLI Htet and Dras (2025)	L (3)	Burmese	Natively-translated	No Adaptation
NLG	AS	XL-Sum Hasan et al. (2021)	Sum	Burmese	English-adapted	Native re-translation	✓

Table 1: Dataset information for each task in BURMESE-SAN . A checkmark in the Our contribution column indicates direct contributions to the adaptation process.

Competency	Task	Label	# samples
NLU	QA	Span	120
	SA	Positive	200
		Negative	200
		Neutral	200
	TD	Clean	200
		Hate	200

Table 2: Class distribution per task in BURMESE-SAN Benchmark Dataset

Model	MY	CR	NLI	QA	SA	TD	AS	MT
Small Models ( $<$ 14B)
MERaLiON 2 (10B)	10.66 ± 0.16	0.00 ± 0.00	12.61 ± 0.31	0.00 ± 0.00	0.07 ± 0.13	14.79 ± 0.09	19.49 ± 0.27	35.33 ± 0.15
Olmo 2 1124 (13B)	1.84 ± 0.09	0.00 ± 0.00	0.00 ± 0.00	0.00 ± 0.00	0.00 ± 0.00	0.79 ± 0.03	0.00 ± 0.00	1.04 ± 0.03
Olmo 2 1124 (7B)	2.18 ± 0.12	0.00 ± 0.00	0.00 ± 0.00	0.00 ± 0.00	0.00 ± 0.00	1.80 ± 0.04	0.00 ± 0.00	6.44 ± 0.10
Olmo 3 (7B)	3.41 ± 0.11	0.00 ± 0.00	0.00 ± 0.00	0.00 ± 0.00	0.00 ± 0.00	3.66 ± 0.05	14.00 ± 0.14	1.11 ± 0.02
Tulu 3 (8B)	10.95 ± 0.14	0.00 ± 0.00	1.35 ± 0.13	5.89 ± 0.64	0.00 ± 0.00	13.70 ± 0.06	21.48 ± 0.17	37.48 ± 0.10

Table 3: Performance of instruct models on BURMESE-SAN tasks. Best model for each task per size group is bold, second best is underlined.

Small Models ( $<$ 14B)
Model	MY	CR	NLI	QA	SA	TD	AS	MT
Olmo 3 Think (7B)	3.85 ± 0.16	0.00 ± 0.00	0.00 ± 0.00	0.00 ± 0.00	0.00 ± 0.00	5.88 ± 0.06	9.99 ± 0.23	17.61 ± 0.12
SEA-LION v3.5 R (Llama) (8B)	22.08 ± 0.20	0.00 ± 0.00	30.19 ± 0.64	54.00 ± 1.22	0.00 ± 0.00	22.28 ± 0.15	11.02 ± 0.32	48.28 ± 0.32
Qwen 3 (Thinking) (4B)	34.78 ± 0.29	40.78 ± 1.30	31.98 ± 0.70	60.37 ± 0.96	31.82 ± 0.53	35.77 ± 0.28	16.29 ± 0.40	56.00 ± 0.11
Qwen 3 (Thinking) (8B)	37.12 ± 0.22	55.80 ± 0.95	33.45 ± 0.63	65.19 ± 1.23	40.18 ± 0.37	40.22 ± 0.22	18.28 ± 0.18	59.53 ± 0.12
Medium Models (14B–32B)

Table 4: Performance of Reasoning models on BURMESE-SAN tasks. Best model for each task per size group is bold, second best is underlined.

Model	MY	CR	NLI	QA	SA	TD	AS	MT
Opus 4.1 (2025-08-05)	45.10 ± 0.15	71.57 ± 0.35	0.00 ± 0.00	0.00 ± 0.00	48.35 ± 0.19	45.66 ± 0.08	24.69 ± 0.09	87.52 ± 0.03
Sonnet 4	40.45 ± 0.30	24.33 ± 0.13	86.01 ± 0.03	0.00 ± 0.00	47.82 ± 0.25	43.22 ± 0.15	68.10 ± 0.67	0.00 ± 0.00
Gemini 2 Flash	59.23 ± 0.16	76.58 ± 0.43	41.30 ± 0.43	79.81 ± 0.33	41.48 ± 0.24	57.78 ± 0.12	22.09 ± 0.09	88.23 ± 0.04
Gemini 2.5 Flash	69.34 ± 0.16	83.58 ± 0.40	61.07 ± 0.46	84.15 ± 0.53	39.60 ± 0.21	66.16 ± 0.10	24.11 ± 0.14	89.49 ± 0.02
Gemini 2.5 Pro	72.35 ± 0.18	85.75 ± 0.38	67.55 ± 0.43	85.48 ± 0.53	45.56 ± 0.30	68.74 ± 0.11	24.21 ± 0.12	90.22 ± 0.02
GPT 4.1 (2025-04-14)	55.80 ± 0.18	68.80 ± 0.37	42.80 ± 0.38	77.15 ± 0.31	50.24 ± 0.27	54.05 ± 0.11	21.37 ± 0.24	79.73 ± 0.08

Table 5: Performance of Commercial models on BURMESE-SAN tasks. Best model for each task per size group is bold, second best is underlined.

Figure 2: Left: Comparison of original models against SEA-fine-tuned variants, and Right: SEA-LION models with their quantized versions - NVIDIA FP4 (NVFP4) and Dynamic FP8 (DynFP8).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Dataset	License
QA	Belebele Bandarkar et al. (2024)	CC BY-NC 4.0
SA	GKLMIP-mya Jiang et al. (2021)	Unknown
TD	myHateSpeech Kyaw et al. (2024)	CC BY-NC-SA 4.0
CR	Balanced COPA Kavumba et al. (2019)	CC BY 4.0
NLI	myXNLI Htet and Dras (2025)	CC BY-NC 4.0
AS	XL-Sum Hasan et al. (2021)	CC BY-NC-SA 4.0

Appendix

Table 6: Licenses for all datasets included in BURMESE-SAN , with corresponding references.

Figure 3: Acceptable and Not Acceptable Grammar Errors in the Dataset.

Figure 4: Different Types of Spelling Errors.

Criteria	Tasks	Definition
Completeness	All	Is all the intended information from the source instruction retained in the translated instruction?
Fluency	All	Is the output grammatically correct and natural in Burmese?
Sensibility	All	Is the translation logical and sensible based on the context of the original instruction?
Faithfulness	Translation	Does the translated instruction stay true to the meaning of the original instruction?
Relevance	Summarization	Does the output (summary) contain only the most important content?
Coherence	Summarization	Is the output (summary) logically structured and easy to follow given the context of the original instruction?

Appendix

Table 7: Quality Evaluation Criteria for BURMESE-SAN

Task	Criterion	Joint Agreement (%)
QA	Completeness	100.00
	Fluency	97.50
	Sensibility	93.33
MT	Completeness	100.00
	Fluency	99.76
	Sensibility	99.88

Appendix

Table 8: Joint agreement (%) for each evaluation criterion across tasks requiring (re-)translation. Scores are before revision by native speakers.

Figure 5: Examples of variation in Burmese translations by native speakers. (a) Differences in technical term usage, transliteration, and word order. (b) Differences in particle choice and formality.

Figure 6: Example Data Samples for each task in BURMESE-SAN .

Panel of Figure 7 — Figure 7: Prompt templates used for BURMESE-SAN . English prompt versions are also provided.

Models	Source Link	No. params
Instruction Tuning Models
ERNIE 4.5	https://huggingface.co/baidu/ERNIE-4.5-300B-A47B-PT	300B MoE
Qwen 3 A22B	https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507	235B MoE
Llama 4 Maverick	https://huggingface.co/meta-llama/Llama-4-Maverick-17B-128E-Instruct	400B MoE
DeepSeek V3.1	https://huggingface.co/deepseek-ai/DeepSeek-V3.1	671B MoE
SEA-LION v4 (Qwen 3)	https://huggingface.co/aisingapore/Qwen-SEA-LION-v4-32B-IT	32B

Appendix

Table 9: Links and sizes of models evaluated in our current experiments.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

ThaiSafetyBench: Assessing Language Model Safety in Thai Cultural Contexts
Mar 5, 2026Trapoom Ukarapol, Nut Chukamphaeng, Kunat Pipatanakul +1Language Model Safety Evaluation LLM Safety
SCB DataX · Department of Computer Science and Technology, Tsinghua University · SCBX R&D +1
SEA-NLI: Natural Language Inference as a Lens into Southeast Asian Cultural Understanding
Jun 2, 2026Peerawat Chomphooyod, Jian Gang Ngui, Yosephine Susanto +5Multilingual Language Model Evaluation Cross-Cultural Language Model Evaluation
1Chulalongkorn University · 2AI Singapore · 3MBZUAI +1
BenHalluEval: A Multi-Task Hallucination Evaluation Framework for Large Language Models on Bengali
May 29, 2026Shefayat E Shams Adib, Ahmed Alfey Sani, Ekramul Alam Esham +3LLM Reliability Multilingual Language Model Evaluation
Department of Computer Science and Engineering, Islamic University of Technology · Department of Computer Science and Engineering, University of California

Mar 5, 2026J/K move · Enter open · S save

ThaiSafetyBench: Assessing Language Model Safety in Thai Cultural Contexts

Trapoom Ukarapol, Nut Chukamphaeng, Kunat Pipatanakul, Pakhapoom Sarapat

SCB DataX · Department of Computer Science and Technology, Tsinghua University · SCBX R&D+1

Language Model Safety Evaluation LLM Safety LLM Safety Benchmarks LLM Safety Monitoring Cross-Cultural Language Model Evaluation Multilingual Language Model Evaluation

Open paper AskPDF arXiv

The safety evaluation of large language models (LLMs) remains largely centered on English, leaving non-English languages and culturally grounded risks underexplored. In this work, we investigate LLM safety in the context of the Thai language and culture and introduce ThaiSafetyBench, an open-source benchmark comprising 1,954 malicious prompts written in Thai. The dataset covers both general harmful prompts and attacks that are explicitly grounded in Thai cultural, social, and contextual nuances. Using ThaiSafetyBench, we evaluate 24 LLMs, with GPT-4.1 and Gemini-2.5-Pro serving as LLM-as-a-judge evaluators. Our results show that closed-source models generally demonstrate stronger safety performance than open-source counterparts, raising important concerns regarding the robustness of openly available models. Moreover, we observe a consistently higher Attack Success Rate (ASR) for Thai-specific, culturally contextualized attacks compared to general Thai-language attacks, highlighting a critical vulnerability in current safety alignment methods. To improve reproducibility and cost efficiency, we further fine-tune a DeBERTa-based harmful response classifier, which we name ThaiSafetyClassifier. The model achieves a weighted F1 score of 84.4%, matching GPT-4.1 judgments. We publicly release the fine-tuning weights and training scripts to support reproducibility. Finally, we introduce the ThaiSafetyBench leaderboard to provide continuously updated safety evaluations and encourage community participation. - ThaiSafetyBench HuggingFace Dataset: https://huggingface.co/datasets/typhoon-ai/ThaiSafetyBench - ThaiSafetyBench Github: https://github.com/trapoom555/ThaiSafetyBench - ThaiSafetyClassifier HuggingFace Model: https://huggingface.co/typhoon-ai/ThaiSafetyClassifier - ThaiSafetyBench Leaderboard: https://huggingface.co/spaces/typhoon-ai/ThaiSafetyBench-Leaderboard

Abstract

LLM Evaluation

Explore similar work

ThaiSafetyBench: Assessing Language Model Safety in Thai Cultural Contexts

SEA-NLI: Natural Language Inference as a Lens into Southeast Asian Cultural Understanding

BenHalluEval: A Multi-Task Hallucination Evaluation Framework for Large Language Models on Bengali