Organizations: Dept. Computer Science and Information Engineering, National Taiwan Normal University, Taiwan · United Link Co., Ltd., Taiwan · Institute of Computer Science, Academia Sinica, Taiwan
Taiwanese Hakka is a low-resource, endangered language that poses significant challenges for automatic speech recognition (ASR), including high dialectal variability and the presence of two distinct writing systems (Hanzi and Pinyin). Traditional ASR models often encounter difficulties in this context, as they tend to conflate essential linguistic content with dialect-specific variations across both phonological and lexical dimensions. To address these challenges, we propose a unified framework grounded in the Recurrent Neural Network Transducers (RNN-T). Central to our approach is the introduction of dialect-aware modeling strategies designed to disentangle dialectal "style" from linguistic "content", which enhances the model's capacity to learn robust and generalized representations. Additionally, the framework employs parameter-efficient prediction networks to concurrently model ASR (Hanzi and Pinyin). We demonstrate that these tasks create a powerful synergy, wherein the cross-script objective serves as a mutual regularizer to improve the primary ASR tasks. Experiments conducted on the HAT corpus reveal that our model achieves 57.00% and 40.41% relative error rate reduction on Hanzi and Pinyin ASR, respectively. To our knowledge, this is the first systematic investigation into the impact of Hakka dialectal variations on ASR and the first single model capable of jointly addressing these tasks.
Figures & tables
Figure 1: Overall architecture of the proposed method, including an RNN-T architecture with multiple joint networks and prediction networks for Hanzi(H) and Pinyin(P) ASR task, and a dialect-aware module.
Dialect
Hanzi
Pinyin
Sixian
佢發譴哩
gi11 fad2 kien31 le24
Nansixian
佢發譴吔
i11 fad2 kien31 e24
Hailu
佢發閼咧
gi55 bod5 ad5 le53
Table 1: Example sentences expressing the same meaning, “He is angry”, across three Taiwanese Hakka dialects. The corresponding Mandarin translation is “他生氣了”. Notably, the Hanzi forms differ in characters such as “譴”, “閼”, and “吔”, reflecting dialect-specific lexical variation. The Pinyin representations also exhibit distinct syllabic realizations, for instance “kien31” versus “ad5”. Moreover, even when the same Hanzi character is shared across dialects, the corresponding pronunciations often diverge, highlighting the phonetic variation inherent to Taiwanese Hakka.
Strategy
Target Sequence
PSC
佢發閼咧 d
PRSC
d 佢發閼咧
TIC
佢 d 發 d 閼 d 咧 d
Table 2: Illustration of example target sequence in each dialect conditioning strategy. d means the dialect ID.
Split
Duration (hours)
# Utterances
Train
73.91
36,867
Development
3.95
2,330
Test
4.46
2,439
Total
82.32
41,636
Table 3: Statistics of the dialect-balanced subset constructed from the HAT corpus, including total duration and number of utterances in each split.
Dialect
Hanzi (CER %)
Pinyin (SER %)
Sixian
10.61
8.56
NanSixian
8.35
10.60
Hailu
17.27
11.25
Mix
6.07
8.29
Table 4: Comparison of Hanzi CER and Pinyin SER for single-task models trained on individual dialects. “Mix” denotes training with data combined from all three dialects, resulting in the best overall performance across both tasks.
Method
Hanzi
Pinyin
Dialect
Eval.
CER %
Rel. %
SER %
Rel. %
Acc. %
VANH
6.07
–
–
–
–
VANP
–
–
8.29
–
–
MTLHP
5.57
8.23
7.99
3.61
–
Dialect-Aware Modeling
w/ ADC
5.29
12.80
7.85
5.31
100.00
Table 5: Comparison of CER of Hanzi and SER of Pinyin of the model with each dialect integration strategy. VAN (Vanilla) denotes models trained on a single task, while MTL denotes joint training on multiple tasks. H and P refer to the Hanzi, Pinyin tasks, respectively. Rel.% indicates relative improvement over the corresponding VAN baseline. All strategies listed under "Dialect-Aware Modeling" and "Dialect Conditioning" are built upon the MTLHP model.
Figure 2: Effect of dialect Acc on ASR performance of our method on the test set. Performance is measured using CER (%) for Hanzi and SER (%) for Pinyin.
Model
Params (M)
RTFx
VANH
71.73
1101.63
MTLHP
w/ ADC , DII and TIC
78.71
1309.86
Table 6: Comparison of model parameters (in millions) and RTFx between the single-task baseline and our proposed method. The RTFx represents the mean of 5 independent runs on a single RTX-3090 GPU.
Low-resource automatic speech recognition (ASR) continues to pose significant challenges, primarily due to the limited availability of transcribed data for numerous languages. While a wealth of spoken content is accessible in television dramas and online videos, Taiwanese Hokkien exemplifies this issue, with transcriptions often being scarce and the majority of available subtitles provided only in Mandarin. To address this deficiency, we introduce TG-ASR for Taiwanese Hokkien drama speech recognition, a translation-guided ASR framework that utilizes multilingual translation embeddings to enhance recognition performance in low-resource environments. The framework is centered around the parallel gated cross-attention (PGCA) mechanism, which adaptively integrates embeddings from various auxiliary languages into the ASR decoder. This mechanism facilitates robust cross-linguistic semantic guidance while ensuring stable optimization and minimizing interference between languages. To support ongoing research initiatives, we present YT-THDC, a 30-hour corpus of Taiwanese Hokkien drama speech with aligned Mandarin subtitles and manually verified Taiwanese Hokkien transcriptions. Comprehensive experiments and analyses identify the auxiliary languages that most effectively enhance ASR performance, achieving a 14.77% relative reduction in character error rate and demonstrating the efficacy of translation-guided learning for underrepresented languages in practical applications.
Cheng-Yeh Yang, Chien-Chun Wang, Li-Wei Chen +3
Dept. Computer Science and Information Engineering, National Taiwan Normal University, Taiwan · United Link Co., Ltd., Taiwan · Institute of Computer Science, Academia Sinica, Taiwan
In Taiwanese Hokkien automatic speech recognition (ASR), prior studies often treat tone sandhi as a major challenge under the assumption that models fail to process implicit phonological variations. However, our experiments on Taiwanese Hokkien reveal that speech foundation models actually handle tone sandhi variations effectively, and the real performance bottleneck stems from a localized confusion between these variations and retained citation tones. To address this, we propose T-SANDHI to explicitly decouple surface acoustics from underlying lexical intent on top of a frozen Whisper backbone. Using a lexicon-guided multi-task learning structure driven by text-derived pseudo labels, our lightweight hybrid injection module integrates independent citation and sandhi phonetic streams via dynamic gating. Extensive evaluation on the TAT-MOE corpus and two blind test sets demonstrates that this explicit disentanglement effectively resolves tonal mapping confusion, outperforming baselines with strict parameter efficiency.
Hung-Yang Sung, Chien-Chun Wang, Tien-Hong Lo +3
Department of Computer Science and Information Engineering, National Taiwan Normal University, Taiwan · E.SUN Financial Holding Co., Ltd., Taiwan · EZAI, Taiwan
Vietnamese exhibits substantial dialectal phonetic variation across Northern, Central, and Southern regions, where identical lexical items may be realized with markedly different pronunciations. Such variation poses challenges for automatic speech recognition (ASR) and remains difficult to model computationally due to the complex relationship between Vietnamese orthography and phonology. Existing approaches typically address dialect variability at the word level, assuming dialect-invariant mappings between spelling and pronunciation, which limits their ability to capture systematic phonetic differences. We propose a dialect-aware phonetic framework that explicitly models Vietnamese phonological structure and dialectal variation at both the vocabulary and decoding levels. The framework introduces a phonetic vocabulary that decomposes each syllable into structured phonetic components and maps them to dialect-specific IPA representations, together with a phonetic-structure decoder that jointly predicts these components. Experiments on the UIT-ViMD, a only-available dataset for multi-dialect in Vietnamese, show that the proposed approach outperforms various pre-trained baselines, especially matches the performance of the strongest pretrained wav2ve2-base-vi-250h across dialects while using substantially fewer parameters and no external pretraining. Code for experimental reproducibility will be publicly available upon the acceptance of this paper.
Quan Ngoc Hoang, Long Hoang Huu Nguyen, Nghia Hieu Nguyen +2
4Vietnam National University, Ho Chi Minh city, Vietnam