Organizations: Dept. Computer Science and Information Engineering, National Taiwan Normal University, Taiwan · United Link Co., Ltd., Taiwan · Institute of Computer Science, Academia Sinica, Taiwan
Low-resource automatic speech recognition (ASR) continues to pose significant challenges, primarily due to the limited availability of transcribed data for numerous languages. While a wealth of spoken content is accessible in television dramas and online videos, Taiwanese Hokkien exemplifies this issue, with transcriptions often being scarce and the majority of available subtitles provided only in Mandarin. To address this deficiency, we introduce TG-ASR for Taiwanese Hokkien drama speech recognition, a translation-guided ASR framework that utilizes multilingual translation embeddings to enhance recognition performance in low-resource environments. The framework is centered around the parallel gated cross-attention (PGCA) mechanism, which adaptively integrates embeddings from various auxiliary languages into the ASR decoder. This mechanism facilitates robust cross-linguistic semantic guidance while ensuring stable optimization and minimizing interference between languages. To support ongoing research initiatives, we present YT-THDC, a 30-hour corpus of Taiwanese Hokkien drama speech with aligned Mandarin subtitles and manually verified Taiwanese Hokkien transcriptions. Comprehensive experiments and analyses identify the auxiliary languages that most effectively enhance ASR performance, achieving a 14.77% relative reduction in character error rate and demonstrating the efficacy of translation-guided learning for underrepresented languages in practical applications.
Figures & tables
Figure 1: Illustration of the Taiwanese Hokkien drama subtitles. (a) A scene with spoken Taiwanese Hokkien and existing Mandarin subtitles enclosed in a blue box 2 2 2 The meaning of the subtitle is “How could he possibly get involved in such a thing?” in English. Images were adapted from publicly available content by Formosa Television Co., Ltd. for educational use. . (b) The outcome of our framework, showing automatically generated Taiwanese Hokkien subtitles, highlighted with a red box, aligned with the Mandarin subtitles.
Split
Duration (hours)
# Utterances
Train
27.51
50,984
Test
2.79
4,859
Total
30.30
55,843
Table 1: Statistics of YT-THDC, including total duration and number of utterances in each split.
Figure 2: The architecture of the proposed TG-ASR, which leverages our novel parallel gated cross-attention (PGCA) mechanism to integrate multilingual translated transcription inputs for improved knowledge transfer in ASR.
ID
Aux. Lang.
CER %
Rel. %
A0
-
13.40
-
A1
Mandarin (GT)
11.87
11.42
A2
Hindi
13.17
1.72
A3
English
13.10
2.24
A4
French
12.98
3.13
A5
Spanish
12.84
4.18
Table 2: CERs and relative reductions (Rel.) on YT-THDC using different auxiliary languages (aux. lang.), where Mandarin serves as the ground-truth (GT) subtitle reference provided in the corpus.
ID
Configuration
CER %
A6
Full PGCA
11.42
A7
w/o tanh Gating
11.46
A8
Sequential Attention
11.60
A9
Shared Attention
12.00
A10
Addition
27.68
A11
Concatenation
24.09
Table 3: Ablation study on the PGCA mechanism.
ID
Aux. Lang.
CER %
Proximity
Tai.
Man.
A1
Mandarin (GT)
11.87
0.905
-
A2
Hindi
13.17
0.854
0.879
A3
English
13.10
0.552
0.590
A4
French
12.98
0.821
0.847
A5
Spanish
12.84
0.843
0.873
Table 4: CERs and language proximity between each auxiliary language translation and transcriptions of Taiwanese Hokkien (Tai.) subtitles of Mandarin (Man.) on YT-THDC.
Figure 3: CERs on YT-THDC as the number of auxiliary languages increases.
Figure 4: Average activation values of the tanh gating mechanism across auxiliary languages, extracted from a representative decoder layer.
ID
Model
CER %
A6
SeamlessM4T
11.42
A12
NLLB
11.52
Table 5: Comparison of ASR performance using different translation models.
Figure 5: Visualization of the cross-lingual attention weights from a representative PGCA layer, mapping source Mandarin (Man.) tokens to target Taiwanese Hokkien (Tai.) tokens.
Strategy
# Auxiliary Language ( k )
1
2
3
4
5
CER
11.87
11.42
11.50
11.53
11.59
Proximity
11.87
11.56
11.49
11.44
11.59
Gating Value
11.87
11.42
11.50
11.44
11.59
Table 6: Comparison of ASR performance (CER %) using different strategies for selecting the top- k auxiliary languages.
Taiwanese Hakka is a low-resource, endangered language that poses significant challenges for automatic speech recognition (ASR), including high dialectal variability and the presence of two distinct writing systems (Hanzi and Pinyin). Traditional ASR models often encounter difficulties in this context, as they tend to conflate essential linguistic content with dialect-specific variations across both phonological and lexical dimensions. To address these challenges, we propose a unified framework grounded in the Recurrent Neural Network Transducers (RNN-T). Central to our approach is the introduction of dialect-aware modeling strategies designed to disentangle dialectal "style" from linguistic "content", which enhances the model's capacity to learn robust and generalized representations. Additionally, the framework employs parameter-efficient prediction networks to concurrently model ASR (Hanzi and Pinyin). We demonstrate that these tasks create a powerful synergy, wherein the cross-script objective serves as a mutual regularizer to improve the primary ASR tasks. Experiments conducted on the HAT corpus reveal that our model achieves 57.00% and 40.41% relative error rate reduction on Hanzi and Pinyin ASR, respectively. To our knowledge, this is the first systematic investigation into the impact of Hakka dialectal variations on ASR and the first single model capable of jointly addressing these tasks.
An-Ci Peng, Kuan-Tang Huang, Tien-Hong Lo +3
Dept. Computer Science and Information Engineering, National Taiwan Normal University, Taiwan · United Link Co., Ltd., Taiwan · Institute of Computer Science, Academia Sinica, Taiwan
Low-resource ASR remains difficult because scarce transcripts provide limited supervised evidence for target-side generation. To address this gap, we propose SAMA-ASR, a lightweight adapter mechanism that augments the decoder with semantic anchors from auxiliary translations and an acoustic anchor from speech; in principle, the mechanism can be applied to similar encoder--decoder multitask speech models. Through cross-modal adaptation, SAMA-ASR conditions decoder states on translation-derived semantic embeddings and a speech embedding, combining utterance-level meaning with speech-grounded evidence before token prediction. At evaluation time, these semantic anchors can be generated automatically by an upstream speech-to-text translator rather than supplied as oracle translations. Experiments on two 30-hour datasets covering the low-resource Sinitic varieties Taiwanese Hokkien and Hakka show that SAMA-ASR improves over acoustic, prior prompt-based, and semantic-only translation-guided baselines and remains effective in practical automatic semantic-anchor settings; translator-capacity analyses show that useful semantic anchors can be produced by a compact ST model.
Kuan-Tang Huang, Cheng-Yeh Yang, Chien-Chun Wang +3
National Taiwan Normal University, Taiwan · EZAI, Taiwan · Academia Sinica, Taiwan
Speech technologies have advanced rapidly and serve diverse populations worldwide. However, many languages remain underrepresented due to limited resources. In this paper, we introduce TaigiSpeech, a real-world speech intent dataset in Taiwanese Taigi (aka Taiwanese Hokkien/Southern Min), which is a low-resource and primarily spoken language. The dataset is collected from older adults, comprising 21 speakers with a total of 3k utterances. It is designed for practical intent detection scenarios, including healthcare and home assistant applications. To address the scarcity of labeled data, we explore two data mining strategies with two levels of supervision: keyword match data mining with LLM pseudo labeling via an intermediate language and an audio-visual framework that leverages multimodal cues with minimal textual supervision. This design enables scalable dataset construction for low-resource and unwritten spoken languages. TaigiSpeech will be released under the CC BY 4.0 license to facilitate broad adoption and research on low-resource and unwritten languages. The project website and the dataset can be found on https://kwchang.org/taigispeech.
Kai-Wei Chang, Yi-Cheng Lin, Huang-Cheng Chou +9
Massachusetts Institute of Technology, USA · National Taiwan University, Taipei, Taiwan · Signal Analysis and Interpretation Laboratory (SAIL), University of Southern California, USA +3