SEAHateCheck: Functional Tests for Detecting Hate Speech in Low-Resource Languages of Southeast Asia

Ri Chi Ng; Aditi Kumaresan; Yujia Hu; Roy Ka-Wei Lee

cs.CLMar 17, 2026

SEAHateCheck: Functional Tests for Detecting Hate Speech in Low-Resource Languages of Southeast Asia

Authors: Ri Chi Ng, Aditi Kumaresan, Yujia Hu, Roy Ka-Wei Lee

Organizations: Singapore University of Technology and Design, Singapore

Abstract

Hate speech detection relies heavily on linguistic resources, which are primarily available in high-resource languages such as English and Chinese, creating barriers for researchers and platforms developing tools for low-resource languages in Southeast Asia, where diverse socio-linguistic contexts complicate online hate moderation. To address this, we introduce SEAHateCheck, a pioneering dataset tailored to Indonesia, Thailand, the Philippines, and Vietnam, covering Indonesian, Tagalog, Thai, and Vietnamese. Building on HateCheck's functional testing framework and refining SGHateCheck's methods, SEAHateCheck provides culturally relevant test cases, augmented by large language models and validated by local experts for accuracy. Experiments with state-of-the-art and multilingual models revealed limitations in detecting hate speech in specific low-resource languages. In particular, Tagalog test cases showed the lowest model accuracy, likely due to linguistic complexity and limited training data. In contrast, slang-based functional tests proved the hardest, as models struggled with culturally nuanced expressions. The diagnostic insights of SEAHateCheck further exposed model weaknesses in implicit hate detection and models' struggles with counter-speech expression. As the first functional test suite for these Southeast Asian languages, this work equips researchers with a robust benchmark, advancing the development of practical, culturally attuned hate speech detection tools for inclusive online content moderation.

Figures & tables

Protected Categories	Indonesia	Malaysia	the Philippines	Singapore	Thailand	Vietnam
Religion	Yes	Yes	Yes	Yes	Yes	Yes
Ethnicity/Race/Origin	Yes	Yes	Yes	Yes	Yes	Yes
Disabilities	No	Yes	Yes	Yes	Yes	Yes
Gender/Orientation	Yes	Yes	Yes	Yes	Yes	Yes
Age	No	No	No	Yes	Yes	Yes
Vulnerable Workers	No	No	No	No	Yes	No

Table 1. Protected categories represented for each country in SEAHateCheck .

		SEAHateCheck					SGHateCheck ( Ng et al., 2024 )
	Metric	ID	TL	TH	VI	Mean	MS	SG	TA	ZH ‡	Mean
Gold Label	# Test cases	8190	8751	8488	10319	8937	NA	NA	NA	NA	NA
	# Hateful template	5511	5902	5681	6998	6023	NA	NA	NA	NA	NA
	# Non-hateful template	2679	2849	2807	3321	2914	NA	NA	NA	NA	NA
	Avg. words	8.3	9.6	NA	13.2	10.4 †	9.5	8.5	7.2	15.6	10.2
	Avg. characters	50.0	55.2	38.8	57.2	50.3	58.9	45.6	62.4	15.6	45.6

Table 2. Summary of SEAHateCheck and SGHateCheck test instances by language and functional test category, including the number of gold and silver cases and their hateful vs. non-hateful splits. Column headers use language abbreviations: ID =Indonesian, TL =Tagalog, TH =Thai, VI =Vietnamese, MS =Malay, SG =Singlish, TA =Tamil, ZH =Mandarin. High quality rate denotes the proportion of high quality annotations, defined in § 2.5 and § 2.6 for Silver Label test cases. NA marks metrics not reported in the source table. † Weighted average words exclude Thai due to missing word segmentation; Thai cells therefore omit word counts. ‡ For Mandarin (ZH), each character is counted as one word (words = characters).

Dataset		Average Quality score
Language	Label	Sentiment	Context	Natural	Target	functional test
Indonesian	Gold Label	0.978	0.993	0.928	NA	NA
Indonesian	Silver Label	0.827	0.910	0.710	0.867	0.470
Tagalog	Gold Label	0.952	0.993	0.998	NA	NA
Tagalog	Silver Label	0.797	0.850	0.820	0.867	0.607
Thai	Gold Label	0.960	0.987	0.986	NA	NA

Table 3. Average scores for different annotation fields for each language.

Dataset	Dataset Name	Language	Year	Collection Method
SEA	id-multi-label-hate-speech-and-abusive-language-detection ( Ibrohim and Budi, 2019 )	Indonesian	2019	Twitter
	Philippine Election-Related Tweets ( Cabasag et al., 2019 )	Tagalog	2019	Twitter
	HateThaiSent ( Maity et al., 2024 )	Thai	2024	—
	ViHSD ( Luu et al., 2021 )	Vietnamese	2021	Facebook, YouTube
SG	HateM ( Maity et al., 2023 )	Malay	2023	—
	COLDataset ( Deng et al., 2022 )	Mandarin	2022	Zhizhu, Weibo

Table 4. Details of the Datasets Used, Including Collection Year and Method

Region	Language	Target Group
SEA	Indonesian	Religion/Creed, Race/Ethnicity, Physical/disability, Gender/Sexual Orientation, Other invective/slander
	Tagalog	Race, Physical, Sex, Disability, Religion, Class, Quality
	Thai	—
	Vietnamese	Aimed at all groups/individuals
	Malay	Race, Ethnicity, National Origin, Caste, Sexual Orientation, Gender, Gender identity, Religious Affiliation, Age, Disability, or Serious Disease
SG	Mandarin	Race, Religion, Sex, or Sexual Orientation

Table 5. Target Groups for Different Languages

	Language	Full Training Dataset			Sampled Training Dataset
	Language	Not Hateful	Hateful	Total	Not Hateful	Hateful	Total
SEA	Indonesian	9,078	1,457	10,535	3,543	1,457	5,000
	Tagalog	5,340	4,660	10,000	2,500	2,500	5,000
	Thai	3,962	2,115	6,077	2,885	2,115	5,000
	Vietnamese	21,490	2,556	24,046	2,500	2,500	5,000
SG	Mandarin	13,003	12,723	25,726	2,500	2,500	5,000

Table 6. Statistics of Hateful and Not Hateful Samples in the Full Dataset and Sampled Subset. Language subsets with poor label distribution are highlighted in red .

Model	SEA				SG
Model	Indonesian	Tagalog	Thai	Vietnamese	Malay	Mandarin	Singlish	Tamil
Ministral	66.53	65.97	68.82	75.15	66.26	64.92	72.22	65.66
Llama3b	67.37	62.80	72.17	74.38	64.29	66.49	69.54	66.75
Llama8b	65.13	59.49	69.09	72.35	66.16	61.66	73.93	59.39
Sealion	72.08	58.38	70.54	74.43	70.85	68.25	71.66	73.42
Seallm	70.27	56.85	70.48	74.28	65.92	66.18	68.64	54.88

Table 7. F1 scores of different non-finetuned models on SEAHateCheck and SGHateCheck High-Quality Test Cases.

Model	SEA				SG
Model	Indonesian	Tagalog	Thai	Vietnamese	Malay	Mandarin	Singlish	Tamil
Ministral	56.49 ( 10.04 )	65.61 ( 0.36 )	66.87 ( 1.95 )	80.05 ( 4.90 )	63.00 ( 3.26 )	60.77 ( 4.15 )	75.33 ( 3.11 )	70.48 ( 4.82 )
Llama3b	57.91 ( 9.46 )	66.21 ( 3.41 )	67.18 ( 4.99 )	78.28 ( 3.90 )	61.62 ( 2.67 )	58.13 ( 8.36 )	65.73 ( 3.81 )	68.96 ( 2.21 )
Llama8b	71.23 ( 6.10 )	67.86 ( 8.37 )	76.16 ( 7.07 )	79.78 ( 7.43 )	71.50 ( 5.34 )	63.94 ( 2.28 )	78.93 ( 5.00 )	65.24 ( 5.85 )
Sealion	76.24 ( 4.16 )	74.04 ( 15.66 )	74.26 ( 3.72 )	82.17 ( 7.74 )	75.04 ( 4.19 )	63.94 ( 4.31 )	82.27 ( 10.61 )	77.64 ( 4.22 )
Seallm	69.73 ( 0.54 )	69.60 ( 12.75 )	71.16 ( 0.68 )	81.75 ( 7.47 )	72.31 ( 6.39 )	66.27 ( 0.09 )	75.16 ( 6.52 )	67.92 ( 13.04 )

Table 8. F1 scores of fine-tuned models on SEAHateCheck and SGHateCheck High-Quality Test Cases, with changes from non-finetuned results in parentheses. Red = decrease, Blue = increase.

Panel of Figure 1 — Figure 1. Accuracy across Functional Tests for Indonesian (left) and Tagalog (right)

Panel of Figure 2 — Figure 2. F1 Score across Protected Categories on Indonesian (left) and Taglog (right)

Model	SEA				SG
Model	Indonesian	Tagalog	Thai	Vietnamese	Malay	Mandarin	Singlish	Tamil
Ministral	76.27	70.42	66.85	81.59	68.97	68.97	66.33	56.93
Llama3b	62.46	60.79	63.05	69.70	61.58	61.58	60.95	68.98
Llama8b	69.77	67.58	68.43	79.44	69.06	69.06	68.70	62.22
Sealion	68.37	64.54	68.87	78.66	67.77	67.77	74.15	64.26
Seallm	58.28	62.20	66.05	70.94	65.42	65.42	67.08	56.75

Table 9. F1 scores of different non-finetuned models on SEAHateCheck and SGHateCheck Silver Test Cases.

Model	SEA				SG
Model	Indonesian	Tagalog	Thai	Vietnamese	Malay	Mandarin	Singlish	Tamil
Ministral	66.34 ( 9.93 )	74.34 ( 3.92 )	71.52 ( 4.67 )	85.37 ( 3.78 )	70.65 ( 1.68 )	69.97 ( 1.00 )	75.68 ( 9.35 )	62.56 ( 5.63 )
Llama3b	70.17 ( 7.71 )	76.57 ( 15.78 )	74.38 ( 11.33 )	85.98 ( 16.28 )	74.92 ( 13.34 )	71.82 ( 10.24 )	80.16 ( 19.21 )	75.19 ( 6.21 )
Llama8b	57.83 ( 11.94 )	72.07 ( 4.49 )	61.43 ( 7.00 )	83.69 ( 4.25 )	68.05 ( 1.01 )	66.90 ( 2.16 )	73.51 ( 4.81 )	68.19 ( 5.97 )
Sealion	57.16 ( 11.21 )	72.25 ( 7.71 )	65.46 ( 3.41 )	83.19 ( 4.53 )	63.63 ( 4.14 )	66.18 ( 1.59 )	70.87 ( 3.28 )	65.63 ( 1.37 )
Seallm	61.09 ( 2.81 )	73.94 ( 11.74 )	73.99 ( 7.94 )	84.88 ( 13.94 )	73.94 ( 8.52 )	71.67 ( 6.25 )	79.69 ( 12.61 )	63.26 ( 6.51 )

Table 10. F1 scores of different fine-tuned models on SEAHateCheck and SGHateCheck Silver Test Cases, with changes from non-finetuned results in parentheses. Red = decrease, Blue = increase.

Panel of Figure 3 — Figure 3. Accuracy across Silver Functional Tests for Indonesian (left) and Tagalog (right)

Panel of Figure 4 — Figure 4. F1 Score across Protected Categories for Silver Indonesian (left) and Taglog (right)

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Country	Legislation/Regulation Consulted
Indonesia	Undang-undang Nomor 1 Tahun 2024 tentang Perubahan Kedua atas Undang-Undang Nomor 11 Tahun 2008 tentang Informasi dan Transaksi Elektronik [Law Number 1 of 2024 concerning Second Amendment to Law Number 11 of 2008 concerning Electronic Information and Transactions] 3 3 3 https://peraturan.bpk.go.id/Details/274494/uu-no-1-tahun-2024
Malaysia	Content Code 2022 4 4 4 https://www.mcmc.gov.my/skmmgovmy/media/General/registers/Content-Code-2022.pdf
the Philippines	The Indigenous Peoples Rights Act of 1997 5 5 5 https://elibrary.judiciary.gov.ph/thebookshelf/showdocs/2/2562 , Safe Spaces Act 6 6 6 https://elibrary.judiciary.gov.ph/thebookshelf/showdocs/2/90094 , Anti-Violence Against Women and Their Children Act of 2004 7 7 7 https://elibrary.judiciary.gov.ph/thebookshelf/showdocs/2/22128 , The Revised Penal Code 8 8 8 https://elibrary.judiciary.gov.ph/thebookshelf/showdocs/28/20426 , Magna Carta for Disabled Persons 9 9 9 https://elibrary.judiciary.gov.ph/thebookshelf/showdocs/2/3140
Singapore	Maintenance of Religious Harmony Act 10 10 10 https://sso.agc.gov.sg/Act/MRHA1990 , the Penal Code’s section 298A 11 11 11 https://sso.agc.gov.sg/Act/PC1871
Thailand	Thailand Civil Law Commission of Computer Related Offences Act (No. 2), BE 2560 (2017), Royal Decree on the Operation of Digital Platform Service Businesses That Are Subject to Prior Notification, B.E. 2565 (2022)
Vietnam	Bộ luật Lao động [Labour Code (2019)] 12 12 12 https://congbao.chinhphu.vn/noi-dung-van-ban-so-45-2019-qh14-30232 , Luật An ninh mạng [Law on Cybersecurity (2018)] 13 13 13 https://congbao.chinhphu.vn/noi-dung-van-ban-so-24-2018-qh14-26894

Appendix

Table 11. Sources of Legislation consulted for defining protected categories.

Figure 5. Functional Test Description and Example for F1 to F15

Figure 6. Functional Tests Description and Example for F16 to F34

Dataset	Language	Localities
SEAHateCheck (Gold and Silver Annotators and Translators)	Indonesian	Jakarta, Surabaya
	Tagalog	Manila, North Luzon, Davao City
	Thai	Bangkok
	Vietnamese	Hanoi, Ho Chi Minh City
SGHateCheck (Silver Annotators only)	Malay	Singapore
	Mandarin	Singapore, Malaysia

Appendix

Table 13. Dialect of target languages spoken by translators and annotators

		Krippendorf’s Alpha
Dataset	Language	Sentiment	Unnatural	Context	Target	Function Test
	Indonesian	0.902	0.126	-0.007	NA	NA
	Tagalog	0.769	0.049	0.082	NA	NA
	Thai	0.801	0.111	0.047	NA	NA
SEAHateCheck Gold Label	Vietnamese	0.871	0.065	0.050	NA	NA
	Indonesian	0.742	0.139	0.152	0.575	0.574

Appendix

Table 15. Inter-annotator agreement for each dataset and annotation field (See section XXX). Cells are coloured in green have a Krippendorf’s Alpha score of 1, while those in yellow have a score of 0.

Open-Source Models
Model Name	Base Model/SOTA	Params	Release Date	Notes
Llama 3.2 3B Instruct (Llama3b) ( Dubey et al., 2024 )	SOTA	3B	25-Sep-24
Llama 3.1 8B Instruct (Llama8b) ( Dubey et al., 2024 )	SOTA	8B	23-Jul-24
Ministral-8B-Instruct-2410 (Ministral) ( , 2024 )	SOTA	8B	16-Oct-24
SeaLLMs-v3-7B-Chat (Seallm) ( Zhang et al., 2024 )	Qwen2 7B ( Yang et al., 2024 )	7B	30-Jul-24
Llama3.1 8B CPT SEA-LIONv3 Instruct (Sealion) ( Singapore, 2024b )	Llama 3.1 8B Instruct ( Dubey et al., 2024 )	8B	31-Oct-24

Appendix

Table 16. Characteristics of all models tested, divided into open-source and closed-source categories. Abbreviated model names are provided in parentheses next to the full names.

### Instructions:

Hate speech is defined as abuse targeted at a protected group or at its members for being a part of that group. Protected groups are Race, Religion, Ethnicity, Gender and Sexual Identity, Disability and Age.

Using the above definition, return either "Hateful" for text classified as hate speech or "Non-Hateful" for text not classified as hate speech. Do not explain or add additional examples. Only enter Hateful or Non-Hateful and end your response.

### Text:

{text}

### Label:

{label}

Appendix

Table 17. Indonesian

### Instructions:

### Text:

{text}

### Label:

{label}

Appendix

Table 18. Tagalog

### Instructions:

Hate speech is defined as abuse targeted at a protected group or at its members for being a part of that group. Protected groups are Religion, Ethnicity, Gender and Sexual Identity, Disability and Age.

### Text:

{text}

### Label:

{label}

Appendix

Table 19. Thai

### Instructions:

Hate speech is defined as abuse targeted at a protected group or at its members for being a part of that group. Protected groups are Religion, Ethnicity, Gender, HIV Status and Sexual Identity, Disability and Age.

### Text:

{text}

### Label:

{label}

Appendix

Table 20. Vietnamese

### Instructions:

Hate speech in Singapore is defined as abuse targeted at a protected group or at its members for being a part of that group. Protected groups are Race, Religion, Country of Origin, Gender and Sexual Identity, Disability, and Age.

### Text:

{text}

### Label:

{label}

Appendix

Table 21. All Singaporean Languages

Func. Class	Functionality	G. Label	# of Annotated Cases
		G. Label	ID	TG	TH	VN	SS	MS	ZH	TA
Derogation	F1 : Expression of strong negative emotions (explicit)	H	-	-	-	-	140	126	140	140
	F2 : Description using very negative attributes (explicit)	H	-	-	-	-	84	112	112	210
	F3 : Dehumanisation (explicit) (explicit)	H	-	-	-	-	131	132	126	146
	F4 : Implicit derogation	H	-	-	-	-	303	140	139	140
Threat. language	F5 : Direct threat (explicit)	H	-	-	-	-	131	119	140	140

Appendix

Table 22. Number of test-cases annotated in SGHateCheck across functionalities. Also shown in this table is the functional class which the functionalities belong to, its functionality number and gold labels.

Panel of Figure 7 — Figure 7. Accuracy across Functional Tests for Thai (left) and Vietnamese (right)

Panel of Figure 8 — Figure 8. Accuracy across Functional Tests for Malay (left) and Mandarin (right)

Panel of Figure 9 — Figure 9. Accuracy across Functional Tests for Singlish (left) and Tamil (right)

Panel of Figure 10 — Figure 10. Accuracy across Silver Functional Tests for Thai (left) and Vietnamese (right)

Panel of Figure 11 — Figure 11. Accuracy across Silver Functional Tests for Malay (left) and Mandarin (right)

Panel of Figure 12 — Figure 12. Accuracy across Silver Functional Tests for Singlish (left) and Tamil (right)

Panel of Figure 13 — Figure 13. F1 Score across Protected Categories for Thai (left) and Vietnamese (right)

Panel of Figure 14 — Figure 14. F1 Score across Protected Categories for Malay (left) and Mandarin (right)

Panel of Figure 15 — Figure 15. F1 Score across Protected Categories for Singlish (left) and Tamil (right)

Panel of Figure 16 — Figure 16. F1 Score across Protected Categories for Silver Thai (left) and Vietnamese (right)

Panel of Figure 17 — Figure 17. F1 Score across Protected Categories for Silver Malay (left) and Mandarin (right)

Panel of Figure 18 — Figure 18. F1 Score across Protected Categories for Silver Singlish (left) and Tamil (right)

Hate Speech Detection

Hate Speech Detection identifies hateful or abusive language, its targets and intent, to support safer and more accountable content moderation.

See all 100 papers

Beyond Benchmarks: Exposing the Hidden Crisis in Bangla Hate Speech Detection
Jul 13, 2026Faria Afrin Tisha, Fariya Tabassum, Hafsa Binte Kibria +2Hate Speech Detection Low-Resource Language Processing
Department of Electrical and Computer Engineering, Rajshahi University of Engineering and Technology, Rajshahi, 6204, Bangladesh. · Department of Computer Science, University of York, Deramore Lane, Heslington, York YO10 5GH, UK.
Efficient Adaptation of LLMs for Hate Speech Detection in Low-Resource Languages: A Comparative Study on Roman Urdu
Aug 6, 2026Toneema Zubair, Muhammad Junaid Asif, Faisal Kamiran +2LLM Evaluation Hate Speech Detection
Department of Computer Sciences Information Technology University Lahore (54000), Pakistan · Artificial Intelligence Technology Centre National Centre for Physics Islamabad (44000), Pakistan
DeepSEA: A Multilingual Evaluation Benchmark for Southeast Asia
Oct 19, 2026Jiuzhou Han, Abhishek Bijay Mishra, Trevor Cohn +1Multilingual Language Model Evaluation Multilingual Language Models
Google Research

Jul 13, 2026J/K move · Enter open · S save

Beyond Benchmarks: Exposing the Hidden Crisis in Bangla Hate Speech Detection

Faria Afrin Tisha, Fariya Tabassum, Hafsa Binte Kibria, Md. Nahiduzzaman+1

Department of Electrical and Computer Engineering, Rajshahi University of Engineering and Technology, Rajshahi, 6204, Bangladesh. · Department of Computer Science, University of York, Deramore Lane, Heslington, York YO10 5GH, UK.

Hate Speech Detection Low-Resource Language Processing

Open paper AskPDF arXiv

The spread of hate speech (HS) across different social media platforms (SMPs) poses a major concern for online safety and ethical moderation. Automatic detection of HS remains a challenging task, especially in under-resourced languages like Bangla, due to cultural context, implicit expressions, and informal linguistic patterns. This study aimed to expose the crisis of Bangla HS detection systems by diagnosing how and why benchmark-trained models fail to identify implicit, context-dependent HS. Six architectures (FastText + CNN, FastText + LSTM, FastText + BiLSTM, BanglaBERT, BanglaBERT + CNN, and BanglaBERT + BiLSTM) were trained on benchmark datasets (about 75,000 posts) and a merged multi-source dataset (about 120,000 posts), then externally validated on an annotated dataset (about 200 posts) collected from Facebook, Twitter, and YouTube, labeled as HS and non-HS, where HS was further categorized as explicit and implicit. BanglaBERT achieved an F1-score of 91.4% on benchmark datasets but declined to 75.3% on the external set and 63.4% for implicit HS involving sarcasm and emojis. The accuracy of FastText + CNN dropped from 78.0% to 51.2% under similar conditions. Emoji-aware preprocessing improved implicit HS detection by up to 12%, whereas emoji removal caused a notable decline in performance (F1: 0.75 to 0.63). Frequent misclassifications in politically charged or satirical comments revealed over-policing risks. This study not only exposes the generalization crisis due to implicit, culturally embedded, and emoji-laden expressions but also underscores the need for developing adaptive, emoji-aware, and culturally grounded frameworks that ensure ethical moderation while preserving freedom of expression. Findings of this study provide insights for researchers, SMPs, and policymakers to design more context-sensitive HS detection systems for low-resource languages.

Abstract

Hate Speech Detection

Explore similar work

Beyond Benchmarks: Exposing the Hidden Crisis in Bangla Hate Speech Detection

Efficient Adaptation of LLMs for Hate Speech Detection in Low-Resource Languages: A Comparative Study on Roman Urdu

DeepSEA: A Multilingual Evaluation Benchmark for Southeast Asia