TW-Sound580K: A Regional Audio-Text Dataset with Verification-Guided Curation for Localized Audio-Language Modeling

Hao-Hui Xie; Ho-Lam Chung; Yi-Cheng Lin; Ke-Han Lu; Wenze Ren; Xie Chen; Hung-yi Lee

cs.SDMar 5, 2026

TW-Sound580K: A Regional Audio-Text Dataset with Verification-Guided Curation for Localized Audio-Language Modeling

Authors: Hao-Hui Xie, Ho-Lam Chung, Yi-Cheng Lin, Ke-Han Lu, Wenze Ren, Xie Chen, Hung-yi Lee

Organizations: National Taiwan University · Shanghai Jiao Tong University

Abstract

Large Audio-Language Models (LALMs) typically struggle with localized dialectal prosody due to the scarcity of specialized corpora. We present TW-Sound580K, a Taiwanese audio-text instruction dataset developed through a Verify-Generate-Critique (VGC) protocol. This pipeline leverages Dual-ASR validation to filter 522K raw clips, subsequently expanding them into 580,000 high-fidelity instruction pairs using a teacher model. The dataset's utility is demonstrated through Tai-LALM, which fine-tunes a DeSTA 2.5-Audio-initialized backbone and incorporates a dynamic Dual-ASR Arbitration strategy to optimize transcription selection during inference. On the TAU Benchmark, Tai-LALM reaches 49.1% accuracy, marking a 6.5% absolute improvement over the zero-shot baseline (42.6% with ASR text conditioning). This confirms that integrating regional corpora with rigorous curation and dynamic arbitration significantly enhances LALM performance on localized speech.

Figures & tables

Figure 1: The proposed framework for TW-Sound580K dataset construction and Tai-LALM fine-tuning, illustrating the DeSTA 2.5-Audio-based localization pipeline.

Panel of Figure 2 — Figure 2: Label occurrence distribution in the TW-Sound580K dataset.

System	Params	Single	Multi	Overall
Gemini 2.5 Pro (Teacher)	–	72.4	73.9	73.0
Gemini 2.5 Flash	–	61.3	63.2	62.1
Qwen2.5-Omni-7B [ 32 ]	7.6B	46.4	46.1	46.3
DeSTA 2.5-Audio [ 24 ]	8.8B	43.3	41.7	42.6
Qwen2-Audio-Instruct [ 33 ]	8.2B	30.3	27.8	29.3
Gemma-3n-E2B-it	4.4B	29.6	25.8	28.0

Table 1: Performance on the TAU Benchmark. Tai-LALM significantly outperforms zero-shot baselines and the naive SFT control.

Panel of Figure 3 — Figure 3: Scaling law analysis demonstrating the efficacy of our localized data pipeline on the TW-Sound580K dataset.

Strategy	SFT Data	ASR Injection	TAU Acc (%)
Zero-shot Baseline	—	None	41.1
Direct SFT (Baseline)	Unfiltered Raw TW-580K+	None	43.8
Single-ASR	Unfiltered Raw TW-580K+	SenseVoice	44.5
Single-ASR(Negative Control)	Unfiltered Raw TW-580K+	Whisper-v3	46.4
Dual-ASR	Unfiltered Raw TW-580K+	Dual-ASR (AC-PPL)	47.5
Ours (VGC Pipeline)	Filtered TW-580K	Dual-ASR (AC-PPL)	49.1

Table 2: Ablation study evaluating the impact of data filtering and inference-time ASR injection strategies on TAU Benchmark accuracy.

Model	LibriSpeech	ESC-50	CREMA-D
	(WER% $\downarrow$ )	(Acc% $\uparrow$ )	(Acc% $\uparrow$ )
DeSTA 2.5-Audio (Zero-shot)	4.71	34.50	75.22
Tai-LALM (Ours)	3.92	33.50	72.28

Table 3: General audio evaluation. Tai-LALM improves LibriSpeech ASR and maintains competitive non-speech performance, confirming no catastrophic forgetting after localized fine-tuning.

Audio-Language Models

Audio-language models connect audio and natural language for understanding, retrieval, dialogue, speech generation, and music or sound analysis.

See all 303 papers

Breeze Taigi: Benchmarks and Models for Taiwanese Hokkien Speech Recognition and Synthesis
Feb 26, 2026Yu-Siang Lan, Chia-Sheng Liu, Yi-Chang Chen +5Speech Processing ASR Evaluation
National Yang Ming Chiao Tung University · Work done at MediaTek Research · MediaTek Research
GlobeAudio: A Multilingual Multicultural Benchmark for Naturalistic Evaluation of Large Audio-Language Models
Jun 6, 2026Ryner Tan, Wenxuan ZhangAudio-Language Model Evaluation Cross-Cultural Language Model Evaluation
Singapore University of Technology and Design
ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge
Jun 23, 2026Jisu Jeon, Seungyeon Jwa, Joosung Lee +6Audio-Language Model Evaluation Pairwise Comparison
Hongik University, South Korea · Seoul National University, South Korea · NAVER Cloud, South Korea +1

Feb 26, 2026J/K move · Enter open · S save

Breeze Taigi: Benchmarks and Models for Taiwanese Hokkien Speech Recognition and Synthesis

Yu-Siang Lan, Chia-Sheng Liu, Yi-Chang Chen, Po-Chun Hsu+4

National Yang Ming Chiao Tung University · Work done at MediaTek Research · MediaTek Research

Speech Processing ASR Evaluation Speech Generation Evaluation Low-Resource TTS Synthesis Low-Resource Speech Recognition

Open paper AskPDF arXiv

Taiwanese Hokkien (Taigi) presents unique opportunities for advancing speech technology methodologies that can generalize to diverse linguistic contexts. We introduce Breeze Taigi, a comprehensive framework centered on standardized benchmarks for evaluating Taigi speech recognition and synthesis systems. Our primary contribution is a reproducible evaluation methodology that leverages parallel Taiwanese Mandarin resources. We provide 30 carefully curated Mandarin-Taigi audio pairs from Taiwan's Executive Yuan public service announcements with normalized ground truth transcriptions. We establish Character Error Rate (CER) as the standard metric and implement normalization procedures to enable fair cross-system comparisons. To demonstrate the benchmark's utility and provide reference implementations, we develop speech recognition and synthesis models through a methodology that leverages existing Taiwanese Mandarin resources and large-scale synthetic data generation. In particular, we fine-tune a Whisper model on approximately 10,000 hours of Taigi synthetic speech data. Our ASR model achieves 30.13% average CER on the benchmark, outperforming existing commercial and research systems. By providing standardized evaluation protocols, diverse training datasets, and open baseline models, we offer a replicable framework with methodologies applicable to various linguistic contexts.

Abstract

Audio-Language Models

Explore similar work

Breeze Taigi: Benchmarks and Models for Taiwanese Hokkien Speech Recognition and Synthesis

GlobeAudio: A Multilingual Multicultural Benchmark for Naturalistic Evaluation of Large Audio-Language Models

ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge