Organizations: Dept. Computer Science and Information Engineering, National Taiwan Normal University, Taiwan · United Link Co., Ltd., Taiwan · Institute of Computer Science, Academia Sinica, Taiwan
Taiwanese Hakka is a low-resource, endangered language that poses significant challenges for automatic speech recognition (ASR), including high dialectal variability and the presence of two distinct writing systems (Hanzi and Pinyin). Traditional ASR models often encounter difficulties in this context, as they tend to conflate essential linguistic content with dialect-specific variations across both phonological and lexical dimensions. To address these challenges, we propose a unified framework grounded in the Recurrent Neural Network Transducers (RNN-T). Central to our approach is the introduction of dialect-aware modeling strategies designed to disentangle dialectal "style" from linguistic "content", which enhances the model's capacity to learn robust and generalized representations. Additionally, the framework employs parameter-efficient prediction networks to concurrently model ASR (Hanzi and Pinyin). We demonstrate that these tasks create a powerful synergy, wherein the cross-script objective serves as a mutual regularizer to improve the primary ASR tasks. Experiments conducted on the HAT corpus reveal that our model achieves 57.00% and 40.41% relative error rate reduction on Hanzi and Pinyin ASR, respectively. To our knowledge, this is the first systematic investigation into the impact of Hakka dialectal variations on ASR and the first single model capable of jointly addressing these tasks.
Figures & tables
Figure 1: Overall architecture of the proposed method, including an RNN-T architecture with multiple joint networks and prediction networks for Hanzi(H) and Pinyin(P) ASR task, and a dialect-aware module.
Dialect
Hanzi
Pinyin
Sixian
佢發譴哩
gi11 fad2 kien31 le24
Nansixian
佢發譴吔
i11 fad2 kien31 e24
Hailu
佢發閼咧
gi55 bod5 ad5 le53
Table 1: Example sentences expressing the same meaning, “He is angry”, across three Taiwanese Hakka dialects. The corresponding Mandarin translation is “他生氣了”. Notably, the Hanzi forms differ in characters such as “譴”, “閼”, and “吔”, reflecting dialect-specific lexical variation. The Pinyin representations also exhibit distinct syllabic realizations, for instance “kien31” versus “ad5”. Moreover, even when the same Hanzi character is shared across dialects, the corresponding pronunciations often diverge, highlighting the phonetic variation inherent to Taiwanese Hakka.
Strategy
Target Sequence
PSC
佢發閼咧 d
PRSC
d 佢發閼咧
TIC
佢 d 發 d 閼 d 咧 d
Table 2: Illustration of example target sequence in each dialect conditioning strategy. d means the dialect ID.
Split
Duration (hours)
# Utterances
Train
73.91
36,867
Development
3.95
2,330
Test
4.46
2,439
Total
82.32
41,636
Table 3: Statistics of the dialect-balanced subset constructed from the HAT corpus, including total duration and number of utterances in each split.
Dialect
Hanzi (CER %)
Pinyin (SER %)
Sixian
10.61
8.56
NanSixian
8.35
10.60
Hailu
17.27
11.25
Mix
6.07
8.29
Table 4: Comparison of Hanzi CER and Pinyin SER for single-task models trained on individual dialects. “Mix” denotes training with data combined from all three dialects, resulting in the best overall performance across both tasks.
Method
Hanzi
Pinyin
Dialect
Eval.
CER %
Rel. %
SER %
Rel. %
Acc. %
VANH
6.07
–
–
–
–
VANP
–
–
8.29
–
–
MTLHP
5.57
8.23
7.99
3.61
–
Dialect-Aware Modeling
w/ ADC
5.29
12.80
7.85
5.31
100.00
Table 5: Comparison of CER of Hanzi and SER of Pinyin of the model with each dialect integration strategy. VAN (Vanilla) denotes models trained on a single task, while MTL denotes joint training on multiple tasks. H and P refer to the Hanzi, Pinyin tasks, respectively. Rel.% indicates relative improvement over the corresponding VAN baseline. All strategies listed under "Dialect-Aware Modeling" and "Dialect Conditioning" are built upon the MTLHP model.
Figure 2: Effect of dialect Acc on ASR performance of our method on the test set. Performance is measured using CER (%) for Hanzi and SER (%) for Pinyin.
Model
Params (M)
RTFx
VANH
71.73
1101.63
MTLHP
w/ ADC , DII and TIC
78.71
1309.86
Table 6: Comparison of model parameters (in millions) and RTFx between the single-task baseline and our proposed method. The RTFx represents the mean of 5 independent runs on a single RTX-3090 GPU.
Dept. Computer Science and Information Engineering, National Taiwan Normal University, Taiwan · United Link Co., Ltd., Taiwan · Institute of Computer Science, Academia Sinica, Taiwan
Department of Computer Science and Information Engineering, National Taiwan Normal University, Taiwan · E.SUN Financial Holding Co., Ltd., Taiwan · EZAI, Taiwan