DeepSEA: A Multilingual Evaluation Benchmark for Southeast Asia

Jiuzhou Han; Abhishek Bijay Mishra; Trevor Cohn; Shubham Mittal

cs.CLOct 19, 2026

DeepSEA: A Multilingual Evaluation Benchmark for Southeast Asia

Authors: Jiuzhou Han, Abhishek Bijay Mishra, Trevor Cohn, Shubham Mittal

Organizations: Google Research

Abstract

While language models have demonstrated remarkable capabilities, their performance remains disproportionately skewed towards high-resource languages. This disparity is particularly acute for the 700 million inhabitants of Southeast Asia, where vast linguistic diversity remains under-represented in existing benchmarks, often suffering from limited language coverage or narrow task scopes. To address this, we introduce DeepSEA, a comprehensive benchmark constructed from established datasets through professional human translation into 12 Southeast Asian languages, spanning both high-resource (e.g., Indonesian, Thai, Vietnamese) and low-resource (e.g., Burmese, Khmer, Lao) tiers across five distinct tasks: mathematical reasoning, scientific reasoning, question answering, abstractive summarization, and machine translation. Extensive evaluation of frontier proprietary and open-weights models reveals that while frontier models like Gemini-3.7-Flash maintain robustness across resource tiers, smaller models face severe performance degradation on low-resource languages. Crucially, models perform better in reasoning and summarization when pivoted through English rather than target languages, exposing a critical dependency on English representations, especially for low-resource languages. Beyond these observations, we provide a cross-lingual error analysis and a cross-script tokenizer fertility analysis. We hope these findings shed light on the development of next-generation inclusive multilingual models.

Figures & tables

Figure 1: Demonstration of various tasks covered in DeepSEA , illustrating specific examples of inputs and outputs.

Panel of Table 1 — Table 1: Linguistic characteristics of the covered languages. The first four languages are classified as high-resource, while the remaining eight are low-resource.

Task	Eval	# high	# low
Mathematical Reasoning	MGSM	1,000	2,000
Scientific Reasoning	IndoMMLU	2,940	5,880
Question Answering	XQuAD	4,760	9,520
Abstractive Summarization	CrossSum	1,558	3,569
Machine Translation	WMT24++	3,988	7,976

Table 2: Number of samples in DeepSEA eval sets, grouped by 4 high and 8 low-resource languages.

Model	Mathematical Reasoning			Scientific Reasoning			Question Answering			Abstractive Summarization			Machine Translation
Model	Accuracy $\uparrow$			Accuracy $\uparrow$			F1 $\uparrow$			ChrF $\uparrow$			MetricX-24 $\downarrow$
Language	en	high	low	en	high	low	en	high	low	en	high	low	high	low
Gemini-3.7-Flash	99.2	96.5	96.4	86.5	86.3	86.1	87.9	85.2	81.7	37.4	36.4	36.2	2.4	3.5
Gemini-3-Pro	98.0	95.6	95.4	85.0	85.3	84.3	85.6	81.2	77.6	37.0	35.9	35.4	2.5	3.7
Gemini-3-Flash	98.8	95.7	95.7	85.7	85.2	84.5	87.2	83.6	79.5	36.9	36.2	36.2	2.5	3.7
GPT-5.6-Luna	96.4	93.3	92.8	85.3	84.5	83.3	73.8	68.1	65.8	33.7	33.8	33.2	2.4	3.5

Table 3: Performance of selected models across different tasks, with languages split among English, high- and low-resource groups as defined in Table 1 . Bold values indicate the best overall performance, while underlined values indicate the best performance among open-weights models.

Figure 2: Frontier models maintain robustness across resource tiers, while open-weights models show severe degradation on low-resource languages. Gemini-3-Flash and Gemma-3-27B are denoted by orange diamonds and blue circles, respectively. The dotted line separates high- and low-resource languages.

Figure 3: Reasoning in English yields consistently higher accuracy than native-language reasoning, especially for low-resource languages. The two left panels compare English and native CoT on reasoning tasks, while the right panel compares direct and English-pivot summarisation.

Figure 4: Radar chart illustrating the answer presence rate across 12 SEA languages in XQuAD dataset.

Figure 5: Relative token fertility across SEA languages when compared to English. Values represent the ratio of tokenized tokens in the given language to English tokens (baseline 1.0), calculated using three tokenizers. Higher values indicate lower tokenization efficiency.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Existing	Newly Created
XQuAD	vi, th	id, fil, ceb, ilo, jv, km, lo, ms, su, my
CrossSum	th, vi, id, my	fil, ceb, ilo, jv, km, lo, ms, su
IndoMMLU	id	th, vi, fil, ceb, ilo, jv, km, lo, ms, su, my
MGSM	th	vi, id, fil, ceb, ilo, jv, km, lo, ms, su, my
WMT24++	vi, th, id, fil	ceb, ilo, jv, km, lo, ms, su, my

Appendix

Table 4: Overview of the data created and reused from existing collections, showing ISO 639-1 language codes.

Task	Split	Language (code)
Task	Split	English (en)	Thai (th)	Vietnamese (vi)	Indonesian (id)	Filipino (fil)	Malay (ms)	Cebuano (ceb)	Ilocano (ilo)	Javanese (jv)	Sundanese (su)	Khmer (km)	Lao (lo)	Burmese (my)
Mathematical Reasoning (MGSM)	test	250	250	250	250	250	250	250	250	250	250	250	250	250
	dev	8	8	8	8	8	8	8	8	8	8	8	8	8
	train	-	-	-	-	-	-	-	-	-	-	-	-	-
Scientific Reasoning (IndoMMLU)	test	735	735	735	735	735	735	735	735	735	735	735	735	735
	dev	15	15	15	15	15	15	15	15	15	15	15	15	15

Appendix

Table 5: Data statistics in DeepSEA benchmark. The table details the number of instances for each task, broken down by language and data split (test, dev, and train). Note that for the CrossSum task, the splits of th, id, vi, and my languages do not have a one-to-one mapping from the English data.

Model	Mathematical Reasoning			Scientific Reasoning			Question Answering			Abstractive Summarisation			Machine Translation
Model	Accuracy $\uparrow$			Accuracy $\uparrow$			F1 $\uparrow$			ChrF $\uparrow$			MetricX-24 $\downarrow$
Language	en	high	low	en	high	low	en	high	low	en	high	low	high	low
Gemini-2.5-Pro	97.2	94.5	94.0	84.2	83.8	83.5	83.7	79.8	75.6	35.8	34.7	34.2	2.6	3.7
Gemini-2.5-Flash	97.2	92.9	92.0	83.7	82.4	82.2	82.9	78.2	74.7	33.7	34.0	34.1	2.7	3.8
Gemma-2-27B	93.2	87.9	71.4	67.5	64.3	57.1	85.2	76.3	69.2	35.0	32.7	27.7	3.4	9.3
Gemma-2-9B	91.2	81.9	61.2	65.2	61.1	53.2	79.4	61.7	57.1	33.5	30.4	21.2	3.9	11.6

Appendix

Table 6: Performance scores of additional models on DeepSEA , which are divided into English, high- and low-resource non-English language groups.

Figure 6: (Left) Translation Quality Assessment. Auto-MQM scores

\geq 0.80

are considered professional quality, and Human Eval percentages represent the rate of accurate translations. (Right) Specific dimensions that are considered in Auto-MQM evaluation.

Figure 7: Comparison of various models in zero-shot and few-shot (three-shot) settings.

Multilingual Language Model Evaluation

Multilingual Language Model Evaluation measures how reliably language models perform across languages, scripts, dialects, and cultural contexts.

See all 485 papers

SEA-SpeechBench: A Large-Scale Multitask Benchmark for Speech Understanding Across Southeast Asia
Sep 9, 2026Jingyi Liao, Wenyu Zhang, Zhuohan Liu +6Audio-Language Model Evaluation ASR Evaluation
Institute of Advanced Intelligence and Computing, A*STAR · Nanyang Technological University · Center for AI Safety +1
SEA-NLI: Natural Language Inference as a Lens into Southeast Asian Cultural Understanding
Jun 2, 2026Peerawat Chomphooyod, Jian Gang Ngui, Yosephine Susanto +5Multilingual Language Model Evaluation Cross-Cultural Language Model Evaluation
1Chulalongkorn University · 2AI Singapore · 3MBZUAI +1
SEATauBench: Adapting Tool-Agent-User Evaluation Into Low-Resource Southeast Asian Languages
Jun 27, 2026My Chiffon Nguyen, Aulia Adila, Saksorn Ruangtanusak +4LLM Agent Evaluation AI Agent Evaluation
SEACrowd · SCB DataX, SCBX Group · Chulalongkorn University +2

Sep 9, 2026J/K move · Enter open · S save

SEA-SpeechBench: A Large-Scale Multitask Benchmark for Speech Understanding Across Southeast Asia

Jingyi Liao, Wenyu Zhang, Zhuohan Liu, Yingxu He+5

Institute of Advanced Intelligence and Computing, A*STAR · Nanyang Technological University · Center for AI Safety+1

Audio-Language Model Evaluation ASR Evaluation Spoken Language Understanding Speech Processing Speech Translation Evaluation

Open paper AskPDF arXiv

The rapid advancement of audio and multimodal large language models has unlocked transformative speech understanding capabilities, yet evaluation frameworks remain predominantly English-centric, leaving Southeast Asian (SEA) languages critically underrepresented. We introduce SEA-SpeechBench, to the best of our knowledge, the first large-scale multitask benchmark that evaluates speech understanding in 11 SEA languages through 97,194 samples across 99 evaluation sets and 597 hours of curated audio data. Our benchmark comprises 9 diverse tasks across 3 categories: speech processing (automatic speech recognition, speech translation, spoken question answering), paralinguistic analysis (emotion, gender, age, speaker recognition), and temporal understanding, a novel dimension featuring timestamped content queries and temporal localization within extended audio sequences up to 3 minutes. We implement multilingual prompting in both native SEA languages and English to reflect user interactions with audio-language models. Evaluation of leading open-source and proprietary systems reveals marked performance gaps. Across all models, performance remains underwhelming on temporal understanding, emotion recognition, and speech translation. Prompting in low-resource languages such as Burmese and Tamil lags behind English by up to 41 percentage points. Our findings expose critical model limitations and underscore the need for inclusive model development. The SEA-SpeechBench benchmark is available at https://zwenyu.github.io/SEA-SpeechBench/.

Abstract

Multilingual Language Model Evaluation

Explore similar work

SEA-SpeechBench: A Large-Scale Multitask Benchmark for Speech Understanding Across Southeast Asia

SEA-NLI: Natural Language Inference as a Lens into Southeast Asian Cultural Understanding

SEATauBench: Adapting Tool-Agent-User Evaluation Into Low-Resource Southeast Asian Languages