ThaiSafetyBench: Assessing Language Model Safety in Thai Cultural Contexts

Trapoom Ukarapol; Nut Chukamphaeng; Kunat Pipatanakul; Pakhapoom Sarapat

cs.CLMar 5, 2026

ThaiSafetyBench: Assessing Language Model Safety in Thai Cultural Contexts

Authors: Trapoom Ukarapol, Nut Chukamphaeng, Kunat Pipatanakul, Pakhapoom Sarapat

Organizations: SCB DataX · Department of Computer Science and Technology, Tsinghua University · SCBX R&D · SCB 10X

Abstract

The safety evaluation of large language models (LLMs) remains largely centered on English, leaving non-English languages and culturally grounded risks underexplored. In this work, we investigate LLM safety in the context of the Thai language and culture and introduce ThaiSafetyBench, an open-source benchmark comprising 1,954 malicious prompts written in Thai. The dataset covers both general harmful prompts and attacks that are explicitly grounded in Thai cultural, social, and contextual nuances. Using ThaiSafetyBench, we evaluate 24 LLMs, with GPT-4.1 and Gemini-2.5-Pro serving as LLM-as-a-judge evaluators. Our results show that closed-source models generally demonstrate stronger safety performance than open-source counterparts, raising important concerns regarding the robustness of openly available models. Moreover, we observe a consistently higher Attack Success Rate (ASR) for Thai-specific, culturally contextualized attacks compared to general Thai-language attacks, highlighting a critical vulnerability in current safety alignment methods. To improve reproducibility and cost efficiency, we further fine-tune a DeBERTa-based harmful response classifier, which we name ThaiSafetyClassifier. The model achieves a weighted F1 score of 84.4%, matching GPT-4.1 judgments. We publicly release the fine-tuning weights and training scripts to support reproducibility. Finally, we introduce the ThaiSafetyBench leaderboard to provide continuously updated safety evaluations and encourage community participation. - ThaiSafetyBench HuggingFace Dataset: https://huggingface.co/datasets/typhoon-ai/ThaiSafetyBench - ThaiSafetyBench Github: https://github.com/trapoom555/ThaiSafetyBench - ThaiSafetyClassifier HuggingFace Model: https://huggingface.co/typhoon-ai/ThaiSafetyClassifier - ThaiSafetyBench Leaderboard: https://huggingface.co/spaces/typhoon-ai/ThaiSafetyBench-Leaderboard

Figures & tables

Figure 1: Attack Success Rate (ASR) across different LLMs on ThaiSafetyBench .

Risk Area	Type of Harm	Description
Discrimination, Exclusion, Toxicity	Toxic Language (Hate Speech)	Language that promotes hatred, offense, or hostility toward individuals or groups based on protected traits.
	Social Stereotype & Unfair Discrimination	Content that perpetuates biased generalizations or unfair treatment based on gender, ethnicity, or similar attributes.
Human–Chatbot Interaction Harms	Treat Chatbot as a Human	Anthropomorphization of chatbots that may lead to emotional reliance or misinterpretation.
	Mental Health / Overreliance Crisis	Excessive dependence on chatbots for mental health support, risking insufficient or inappropriate intervention.
	Adult Content	Inappropriate or explicit material in human–chatbot interactions.
Information Hazards	Sensitive Information (Organization/Government)	Exposure or leakage of confidential organizational or governmental information.

Table 1: ThaiSafetyBench Dataset Taxonomy

Figure 2: Distribution of samples in ThaiSafetyBench by risk areas and harm subcategories.

Figure 3: Attack Success Rate (ASR) Across Risk Areas

Panel of Figure 4 — Figure 4: Analysis of attack success rate (ASR) across various aspects.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Attack Success Rate (ASR) across Models by GPT-4.1 and Gemini-2.5-pro Evaluators

Figure 6: Qualitative Example in riks area of human-chatbot interaction harms

Figure 7: Qualitative Example in riks area of information hazards

Figure 8: Qualitative Example in riks area of malicious uses

Figure 9: Qualitative Example in riks area of misinformation harms

Figure 10: Qualitative Example in riks area of Thai socio-cultural harm (1)

Figure 11: Qualitative Example in riks area of Thai socio-cultural harm (2)

Figure 12: Qualitative Example in riks area of Thai socio-cultural harm (3)

Language Model Safety Evaluation

Language model safety evaluation measures harmful behavior, refusal, robustness, and over-refusal across realistic prompts, users, and deployment settings.

See all 517 papers

Register Shifts Break LLM Safety: A Bengali Benchmark with Culturally Grounded Harms
Aug 23, 2026Naymul Islam, Nusrat Jahan Lia, Shubhashis Roy Dipta +2LLM Safety Benchmarks Language Model Safety Evaluation
BanglaLLM · Institute of Information Technology, University of Dhaka · University of Maryland, Baltimore County +2
IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asia
Mar 18, 2026Priyaranjan Pattnayak, Sanchari ChowdhuriLanguage Model Safety Evaluation LLM Safety Benchmarks
Oracle America Inc.
Schützen: Evaluating LLM Safety in Bulgarian and German Contexts
Jun 9, 2026Kiril Georgiev, Yuxia Wang, Dimitar Iliyanov Dimitrov +2LLM Safety Benchmarks Multilingual Language Model Evaluation
Sofia University "St. Kliment Ohridski" · INSAIT, Sofia University “St. Kliment Ohridski” · MBZUAI

Aug 23, 2026J/K move · Enter open · S save

Register Shifts Break LLM Safety: A Bengali Benchmark with Culturally Grounded Harms

Naymul Islam, Nusrat Jahan Lia, Shubhashis Roy Dipta, Sabik Bin Sultan+1

BanglaLLM · Institute of Information Technology, University of Dhaka · University of Maryland, Baltimore County+2

LLM Safety Benchmarks Language Model Safety Evaluation LLM Safety Monitoring Cross-Cultural Language Model Evaluation

Open paper AskPDF arXiv

Bengali is the seventh-most-spoken language globally, yet LLM safety evaluation remains overwhelmingly English-centric. We introduce BanglaSafe, a benchmark of 879 Bengali prompts combining 309 natively authored prompts with 570 expert-reviewed prompts, spanning 17 culturally grounded harm categories and five prompting conditions that vary language, writing style, and authority framing. Evaluating 18 frontier LLMs, we find that over half of all responses are unsafe or partially unsafe (53.6%) while 14.7% contains strictly harmful content, and that the strongest observed effect is not the switch from English to Bengali but the choice of writing style within Bengali: the same harmful request phrased as a formal newspaper investigation succeeds 17 percentage points more often than the same request phrased as a casual message, with no adversarial engineering involved. We further show that existing safety classifiers struggle to reliably evaluate Bengali content, with even frontier models failing on nearly half of all cases.

Abstract

Language Model Safety Evaluation

Explore similar work

Register Shifts Break LLM Safety: A Bengali Benchmark with Culturally Grounded Harms

IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asia

Schützen: Evaluating LLM Safety in Bulgarian and German Contexts