SEABED: SouthEast Asian Benchmark for Evaluating Audio Reasoning

Harshit Rajgarhia; Asif Shaik; Rachuri Lokesh; Sushanta Kumar Pani; Abhishek Mukherji

cs.SDSep 18, 2026

SEABED: SouthEast Asian Benchmark for Evaluating Audio Reasoning

Authors: Harshit Rajgarhia, Asif Shaik, Rachuri Lokesh, Sushanta Kumar Pani, Abhishek Mukherji

Organizations: Centific AI Research

Abstract

Modern audio-language models are no longer judged only on what words they can transcribe, but on whether they can reason over what they hear: recovering meaning that lives in tone and prosody, telling dialects and regional languages apart, and resolving ambiguity that the written form leaves open. This capability is now measured by a growing family of audio-reasoning benchmarks, but almost entirely in English and on general-domain audio. Southeast Asia (SEA) is served instead by benchmarks that inherit an English task taxonomy of recognition, translation, and paralinguistic classification, and therefore test whether a model hears SEA speech rather than whether it can reason from it. We introduce SEABED, an audio-first question answering dataset designed to benchmark language and audio reasoning models on SEA speech. SEABED comprises a suite of six audio-reasoning tasks built entirely from real, openly available SEA speech corpora, yielding 5,404 question-answer pairs. We evaluate six frontier and region-specific audio LLMs: even the state-of-the-art model Gemini 3.5 Flash achieves only 50.3% weighted average accuracy. SEABED evaluates not only answer accuracy, but also whether models' stated reasoning is grounded in the audio evidence. A sample of the benchmark data is available here: https://huggingface.co/datasets/CentificAIResearch/SEABED.

Figures & tables

Figure 1: SEABED at a glance: one example item per task (question, options, gold answer).

Figure 2: SEABED construction and evaluation pipeline.

	Multiple-choice						Open-ended
Model	DIALECT	AMBIG	DSC	LONG	ER	SAI	DIALECT	AMBIG	DSC	LONG	ER	SAI	Weighted Avg
Gemini 2.5 Pro	37.5	47.6	68.1	76.1	30.8	36.7	29.1	35.9	44.3	64.6	31.7	30.8	41.6
Gemini 3.5 Flash	41.5	60.5	69.4	80.2	49.4	45.8	34.1	48.5	48.7	66.7	48.8	37.7	50.3
Gemini 3.6 Flash	39.3	60.7	68.3	79.1	40.3	42.7	32.5	49.4	42.5	68.3	41.2	36.6	47.4
GPT-Audio 1.5	18.7	38.9	47.0	50.5	29.4	31.3	22.7	31.3	21.9	50.2	34.9	29.8	32.0
MERaLiON-3-10B	17.9	38.3	42.7	20.3	30.0	38.7	19.5	27.8	28.4	12.9	32.8	29.0	29.4

Table 1: Accuracy on SEABED (%). Task codes follow Table 6 : DIALECT (dialect and language identification), AMBIG (prosodic ambiguity resolution), DSC (dialectal speech comprehension), LONG (long-form audio reasoning), ER (speech emotion recognition), SAI (speech-affective interpretation). Best per column in bold . Weighted Avg is the sample-weighted mean over all items.

Panel of Figure 3 — Figure 3: The reasoning-trace audit at a glance. (a) Genuine audio reasoning rate by task: the share of each model’s correct answers whose reasoning is grounded in the audio. (b) The two audit axes per model: the grounding of reasoning axis, a distribution over all responses, and the failure type axis, a distribution over incorrect answers.

Model	$n$	Acc.	Genuine (of corr.)	Partial (of corr.)	Lucky (of corr.)
Gemini 3.6 Flash	543	48.8	86.0	5.3	8.7
GPT-Audio 1.5	543	30.8	71.3	13.8	15.0
MERaLiON-3-10B	543	28.9	63.1	10.8	26.1

Table 2: How correct answers were reached, on the audited

10\%

sample (percentages).

	Gemini 3.5 Flash			MERaLiON-3-10B
Task	Audio	Trans.	$\Delta$	Audio	Trans.	$\Delta$
DIALECT	37.7	26.4	+11.3	18.6	4.5	+14.1
AMBIG	54.5	14.9	+39.6	33.0	19.8	+13.2
DSC	59.0	82.0	$-$ 23.0	35.5	10.8	+24.7
LONG	55.6	47.6	+8.0	10.4	18.1	$-$ 7.8
Weighted Avg	51.0	41.8	+9.2	27.0	12.4	+14.6

Table 3: Audio-only vs transcript-only accuracy (%) per task, full benchmark, both formats pooled.

\Delta=

Audio

-

Transcript; positive means the audio is needed. Format-split versions are in Appendix E .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Corpus	Lang.	Tasks fed
INDspeech ( Sakti et al., 2008 )	id	DIALECT
SLSCU Thai Dialect ( Suwanbandit et al., 2023 )	th	DIALECT, DSC
STRUCT_AMB_IND ( Widiaputri et al., 2023 )	id	AMBIGUOUS
THAI SER ( AIResearch.in.th, 2021 )	th	ER, SAI
E-SERAVD ( Santoso et al., 2026 )	id	ER, SAI
IndoWaveSentiment ( Bustamin et al., 2024 )	id	ER, SAI

Appendix

Table 4: Source corpora. Languages: id Indonesian, th Thai, ms Malay.

Model	Type	Role	Citation
gemini-3.1-pro-preview	P	QA generation	( Google DeepMind, 2026a )
grok-4.5	P	Judge (open-ended)	( xAI, 2026 )
claude-opus-4.8	P	Reasoning-trace audit	( Anthropic, 2026 )
gemini-2.5-pro	P	Benchmarking	( Comanici et al., 2025 )
gemini-3.5-flash	P	Benchmarking	( Google DeepMind, 2026b )
gemini-3.6-flash	P	Benchmarking	( Google DeepMind, 2026c )

Appendix

Table 5: Model roster. P: proprietary; OS: open source. The QA-generation and judge models are drawn from families different from most benchmarked models, and the judge family differs from the generator family.

#	Task	Phenomenon (why audio matters)
1	Dialect and Language Identification (DIALECT)	Accent and lect identity in ordered multi-dialect audio; content controlled or uninformative
2	Prosodic Ambiguity Resolution (AMBIGUOUS)	Sentence ambiguous in text; prosody selects the intended reading
3	Dialectal Speech Comprehension (DSC)	Content comprehension in dialectal, non-standard speech
4	Long-Form Audio Reasoning (LONG_FORM)	Integration and inference over entire spontaneous conversations
5	Speech Emotion Recognition (ER)	Discrete emotion carried by vocal delivery, not words
6	Speech-Affective Interpretation (SAI)	Joint mood $\times$ energy read from prosody

Appendix

Table 6: The six SEABED tasks and the phenomenon each isolates.

Task	Items	MCQ/OE	Lang.	Audio (h)
DIALECT	1,008	504/504	id, th	1.32
AMBIGUOUS	1,008	504/504	id	2.79
DSC	1,008	504/504	th	1.62
LONG_FORM	364	182/182	th, id, ms	15.66
ER	1,008	504/504	th, id	1.12
SAI	1,008	504/504	th, id	1.11

Appendix

Table 7: Per-task composition. Long-form items are drawn from

122

source conversations totalling

32.16

hours.

Label	Definition
Grounding axis (labels every response)
grounded	cites specific audible content, consistent with the reference transcript, that supports the model’s own answer
partial	some real audio evidence, padded with unsupported leaps
ungrounded	generic or templated justification, pure option elimination, or answer restatement with no audio evidence
contradictory	the stated reasoning points to a different answer than the one given
Failure type axis (labels every wrong answer)

Appendix

Table 8: Audit label definitions for the two reasoning-trace axes (§ 4.3 ). A correct answer labeled grounded counts as genuine audio reasoning; a correct answer labeled ungrounded or contradictory counts as a lucky guess.

				Of correct			Of wrong
Task	Model	$n$	Acc.	Gen.	Par.	Lucky	Per.	Com.	Hal.	Rsn.	Abs.	Fmt.
DIALECT	Gemini 3.6 Flash	100	37.0	45.9	8.1	45.9	1.6	90.5	3.2	0.0	3.2	1.6
	GPT-Audio 1.5	100	21.0	4.8	4.8	90.5	10.1	64.6	8.9	3.8	5.1	7.6
	MERaLiON-3-10B	100	18.0	5.6	16.7	77.8	22.0	65.9	9.8	1.2	0.0	1.2
AMBIG	Gemini 3.6 Flash	102	62.7	100.0	0.0	0.0	36.8	42.1	2.6	13.2	5.3	0.0
	GPT-Audio 1.5	102	33.3	94.1	5.9	0.0	8.8	79.4	2.9	4.4	0.0	4.4

Appendix

Table 9: Per-task audit drill-down on the

10\%

sample. All cells are percentages: Gen./Par./Lucky are shares of correct answers (grounded, partial, ungrounded or contradictory reasoning); Per. through Fmt. are shares of wrong answers (perception, comprehension, hallucination, reasoning, abstention, format). AMBIG stratifies across three variants and two formats, so its

10\%

sample rounds to

n=102

; LONG rests on small samples (

41

items per model; MERaLiON-3-10B has only two correct answers).

Table 13

	Multiple-choice												Open-ended
Model	DIALECT		AMBIG		DSC		LONG		ER		SAI		DIALECT		AMBIG		DSC		LONG		ER		SAI
	ENG	SEA	ENG	SEA	ENG	SEA	ENG	SEA	ENG	SEA	ENG	SEA	ENG	SEA	ENG	SEA	ENG	SEA	ENG	SEA	ENG	SEA	ENG	SEA
Gemini 3.5 Flash	41.3	39.1	69.5	67.3	73.3	68.8	93.3	86.6	44.4	44.4	51.1	48.8	44.4	31.1	51.1	53.3	52.1	58.6	65.7	58.7	52.1	43.4	23.9	28.2
GPT-Audio 1.5	17.3	13.0	28.2	32.6	53.3	50.0	68.8	60.0	33.3	40.9	37.7	40.4	26.6	28.8	26.6	35.5	28.2	21.7	47.2	46.1	34.7	32.6	17.3	28.2
MERaLiON-3-10B	13.0	10.8	32.6	36.9	48.8	48.8	37.7	17.7	28.8	26.6	44.4	35.5	31.1	8.8	28.8	33.3	34.7	23.9	17.7	11.3	32.6	28.2	23.9	26.0

Appendix

Table 12: English (ENG) vs native SEA-language (SEA) question accuracy (%) per task and format,

10\%

stratified sample. Task codes as in Table 6 .

Model	DIALECT			AMBIG			DSC			LONG			ER			SAI
	4	7	10	4	7	10	4	7	10	4	7	10	4	7	10	4	7	10
Gemini 3.5 Flash	60.8	43.4	41.3	71.7	71.7	69.5	84.4	80.0	73.3	95.5	95.5	93.3	57.7	53.3	44.4	62.2	48.8	51.1
GPT-Audio 1.5	45.6	19.5	17.3	56.5	39.1	28.2	71.1	61.9	53.3	80.0	64.4	68.8	51.1	40.9	33.3	44.1	41.0	37.7
MERaLiON-3-10B	28.2	23.9	13.0	58.6	47.8	32.6	55.5	57.7	48.8	47.7	35.5	37.7	55.5	40.0	28.8	66.6	57.7	44.4

Appendix

Table 13: Accuracy (%) vs number of multiple-choice options (4 / 7 / 10),

10\%

stratified sample. Task codes as in Table 6 .

Figure 4: Multiple-choice versus open-ended accuracy by task, pooled over all six models on the full benchmark. Open-ended is lower on every task except speech emotion recognition.

Figure 5: Tonal vs non-tonal language: accuracy on speech emotion recognition and speech-affective interpretation, pooled over all six models and both QA formats (full audio), with $95\%$ Wilson intervals. To compare languages like for like, label composition is controlled: emotion recognition is restricted to the emotion labels present in both languages, while speech-affective interpretation uses identical valence-arousal label sets in both languages by construction. The tonal language performs on par with or better than the non-tonal one in both tasks.

Task	Format	Question	GT	Options	Model	Prediction
Dialect and Language Identification	MCQ	Which segment number is spoken in the Khummuang (Northern) dialect?	(G)	(A) 4th utterance; (B) Cannot be determined from the audio; (C) 5th utterance; …; (G) 2nd utterance; …	Gemini 3.5 Flash	(g) ✓
	MCQ	What is the dialect/accent spoken in the 3rd utterance?	(C)	(A) Betawi; (B) Standard (Umum); (C) Batak; …; (J) Balinese	Gemini 3.5 Flash	(j) ✗
	Open	Which accent comes immediately after the Standard Indonesian (Umum) accent?	Javanese	n/a	GPT-Audio-1.5	Javanese-accented Indonesian ✓
	MCQ	At what position does the Batak accent appear?	(D)	(A) 2nd utterance; (B) None of the segments; (C) The 1st and 4th utterances; (D) 3rd utterance; …; (J) 4th utterance	GPT-Audio-1.5	(j) ✗
	Open	Is the Javanese accent spoken before or after the Batak accent?	before	n/a	MERaLiON-3-10B	Javanese accent is spoken before the Batak accent. ✓
	MCQ	Which accent comes immediately after the Sundanese accent?	(B)	(A) Manado Malay; (B) Batak; (C) Javanese; (D) none (it is the last utterance); …	MERaLiON-3-10B	(d) ✗

Appendix

Table 14: Example model predictions across tasks: the best model of each family (Gemini 3.5 Flash, GPT-Audio-1.5, MERaLiON-3-10B) is shown with one correct and one incorrect prediction per task, over a mix of MCQ and open-ended questions. ✓ indicates correct prediction, ✗ indicates incorrect prediction.

Audio Reasoning

Audio reasoning studies how AI systems infer events, relationships, intent, and context from sound, beyond recognizing words or matching labels.

See all 64 papers

Afrispeech Semantics: Evaluating Audio Semantic Reasoning in Spoken Language Models Across Domains and Accents
May 11, 2026Chibuzor Okocha, Christan GrantAudio-Language Model Evaluation Spoken Language Understanding
University of Florida
SEA-SpeechBench: A Large-Scale Multitask Benchmark for Speech Understanding Across Southeast Asia
Sep 9, 2026Jingyi Liao, Wenyu Zhang, Zhuohan Liu +6Audio-Language Model Evaluation ASR Evaluation
Institute of Advanced Intelligence and Computing, A*STAR · Nanyang Technological University · Center for AI Safety +1
Beyond Accuracy: ARIA-Rubrics for Evaluating Audio Reasoning in Large Audio Language Models
Sep 9, 2026Yupei Li, Qiyang Sun, Mohamed Mady +4Audio-Language Model Evaluation Audio Reasoning
Imperial College London, UK · Technische Universität München München, Germany · Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, AE +2

May 11, 2026J/K move · Enter open · S save

Afrispeech Semantics: Evaluating Audio Semantic Reasoning in Spoken Language Models Across Domains and Accents

Chibuzor Okocha, Christan Grant

University of Florida

Audio-Language Model Evaluation Spoken Language Understanding Audio Reasoning Audio Understanding Audio-Language Models Language Model Robustness

Open paper AskPDF arXiv

Audio language models (ALMs) are increasingly used for speech-based understanding, yet their ability to perform semantic reasoning beyond transcription, Text-to-Audio Retrieval, Captioning, and Question-Answering accuracy remains insufficiently benchmarked. In particular, the effects of accent variation, domain shift, and semantic over-inference on audio reasoning are poorly understood. We evaluate audio language models across five semantic and paralinguistic reasoning tasks: entailment, consistency, plausibility, accent drift, and accent restraint. Collectively, these tasks assess a model's ability to reason over spoken audio as the primary evidence source, including whether a textual hypothesis can be inferred, contradicted, or left undetermined by the audio, whether statements align or conflict with spoken content, whether claims are plausible given the discourse, and whether model predictions remain stable or appropriately constrained across accent variation. These findings highlight critical limitations in current audio reasoning evaluations and hope to provide guidance for more robust and equitable ALM design and assessment

Abstract

Audio Reasoning

Explore similar work

Afrispeech Semantics: Evaluating Audio Semantic Reasoning in Spoken Language Models Across Domains and Accents

SEA-SpeechBench: A Large-Scale Multitask Benchmark for Speech Understanding Across Southeast Asia

Beyond Accuracy: ARIA-Rubrics for Evaluating Audio Reasoning in Large Audio Language Models